System and method for projecting multiple image streams

ABSTRACT

One form of the present invention provides a method of displaying multiple image streams that includes providing a first plurality of image streams, processing the plurality of image streams, thereby generating at least one processed image stream, and projecting the at least one processed image stream onto a non-planar surface with at least one projector, thereby generating a plurality of different projected image streams at a corresponding plurality of different positions on the non-planar surface.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is related to U.S. patent application Ser. No. ______,attorney docket no. 200601919-1, filed on the same date as thisdisclosure, and entitled SYSTEM AND METHOD FOR DISPLAYING IMAGES; U.S.patent application Ser. No. ______, attorney docket no. 200601920-1,filed on the same date as this disclosure, and entitled BLEND MAPS FORRENDERING AN IMAGE FRAME; U.S. patent application Ser. No. ______,attorney docket no. 200601921-1, filed on the same date as thisdisclosure, and entitled SYSTEM AND METHOD FOR GENERATING SCALE MAPS;and U.S. patent application Ser. No. ______, attorney docket no.200601999-1, filed on the same date as this disclosure, and entitledMESH FOR RENDERING AN IMAGE FRAME.

BACKGROUND

Many cameras that capture images have planar image planes to produceplanar images. Planar images captured by such cameras may be reproducedonto planar surfaces. When a viewer views a planar image that has beenreproduced onto a planar surface, the viewer generally perceives theimage as being undistorted, assuming no keystone distortion, even whenthe viewer views the image at oblique angles to the planar surface ofthe image. If a planar image is reproduced onto a non-planar surface(e.g., a curved surface) without any image correction, the viewergenerally perceives the image as being distorted.

Display systems that reproduce images in tiled positions may provideimmersive visual experiences for viewers. While tiled displays may beconstructed from multiple, abutting display devices, these tileddisplays generally produce undesirable seams between the display devicesthat may detract from the experience. In addition, because these displaysystems generally display planar images, the tiled images may appeardistorted and unaligned if displayed on a non-planar surface withoutcorrection. In addition, the display of the images with multiple displaydevices may be inconsistent because of the display differences betweenthe devices.

SUMMARY

One form of the present invention provides a method of displayingmultiple image streams that includes providing a first plurality ofimage streams, processing the plurality of image streams, therebygenerating at least one processed image stream, and projecting the atleast one processed image stream onto a non-planar surface with at leastone projector, thereby generating a plurality of different projectedimage streams at a corresponding plurality of different positions on thenon-planar surface.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A is a block diagram illustrating an image display systemaccording to one embodiment of the present invention.

FIG. 1B is a schematic diagram illustrating a developable surfaceaccording to one embodiment of the present invention.

FIG. 1C is a schematic diagram illustrating the projection of partiallyoverlapping images onto a developable surface without correctionaccording to one embodiment of the present invention.

FIG. 1D is a schematic diagram illustrating the projection of partiallyoverlapping images onto a developable surface with correction accordingto one embodiment of the present invention.

FIGS. 2A-2H are flow charts illustrating methods for geometriccorrection according to one embodiment of the present invention.

FIGS. 3A-3D are schematic diagrams illustrating the generation ofscreen-to-camera triangle meshes according to one embodiment of thepresent invention.

FIGS. 4A-4D are schematic diagrams illustrating the generation ofcamera-to-projector triangle meshes according to one embodiment of thepresent invention.

FIGS. 5A-5B are schematic diagrams illustrating the generation and useof a screen-to-projector a triangle mesh for each projector in an imagedisplay system according to one embodiment of the present invention.

FIGS. 6A-6G are flow charts illustrating methods for photometriccorrection according to one embodiment of the present invention.

FIG. 7 is a schematic diagram illustrating a process of rendering imageframes using photometric maps according to one embodiment of the presentinvention.

FIG. 8 is a block diagram illustrating a process of determining inversetone reproduction functions for each color plane of a projectoraccording to one embodiment of the present invention.

FIGS. 9A and 9B are schematic diagrams illustrating a process ofdetermining blend maps according to one embodiment of the presentinvention.

FIG. 10 is a block diagram illustrating a process of determining offsetmaps according to one embodiment of the present invention.

FIG. 11 is a block diagram illustrating a process of determiningattenuation maps according to one embodiment of the present invention.

FIG. 12 is a block diagram illustrating the processing system shown inFIG. 1A as configured for providing dynamically reconfigurable multiplestream rendering according to one embodiment of the present invention.

FIGS. 13A-13C are diagrams illustrating a simplified representation ofthe simultaneous projection of multiple different streams by the displaysystem shown in FIG. 1A, and the dynamic reconfiguration of theprojected streams according to one form of the present invention.

FIG. 14 is a diagram illustrating a dataflow graph showing theconnections of stream processing modules according to one embodiment ofthe present invention.

FIG. 15 is a diagram illustrating a method of displaying multiple imagestreams according to one embodiment of the present invention.

DETAILED DESCRIPTION

In the following Detailed Description, reference is made to theaccompanying drawings, which form a part hereof, and in which is shownby way of illustration specific embodiments in which the invention maybe practiced. In this regard, directional terminology, such as “top,”“bottom,” “front,” “back,” etc., may be used with reference to theorientation of the Figure(s) being described. Because components ofembodiments of the present invention can be positioned in a number ofdifferent orientations, the directional terminology is used for purposesof illustration and is in no way limiting. It is to be understood thatother embodiments may be utilized and structural or logical changes maybe made without departing from the scope of the present invention. Thefollowing Detailed Description, therefore, is not to be taken in alimiting sense, and the scope of the present invention is defined by theappended claims.

I. Generation and Display of Partially Overlapping Frames onto a Surface

FIG. 1A is a block diagram illustrating an image display system 100according to one embodiment of the present invention. Image displaysystem 100 includes a processing system 101, projectors 112(1) through112(N) where N is greater than or equal to one (collectively referred toas projectors 112), and at least one camera 122. Processing system 101includes image frame buffer 104, frame generator 108, and calibrationunit 124.

Processing system 101 receives streams of image frames 102(1) through102(M) where M is greater than or equal to one (referred to collectivelyas image data 102) using any suitable wired or wireless connectionsincluding any suitable network connection or connections. The streams ofimage frames 102(1) through 102(M) may be captured and transmitted byattached or remote image capture devices (not shown) such as cameras,provided by an attached or remote storage medium such as a hard-drive, aDVD or a CD-ROM, or otherwise accessed from one or more storage devicesby processing system 101.

In one embodiment, a first image capture device captures and transmitsimage frames 102(1), a second image capture device captures andtransmits image frames 102(2), and an Mth image capture device capturesand transmits image frames 102(M), etc. The image capture devices may bearranged in one or more remote locations and may transmit the streams ofimage frames 102(1) through 102(M) across one or more networks (notshown) using one or more network connections.

In one embodiment, the number M of streams of image frames 102 is equalto the number N of projectors 112. In other embodiments, the number M ofstreams of image frames 102 is greater than or less than the number N ofprojectors 112.

Processing system 101 processes the streams of image frames 102(1)through 102(M) and generates projected images 114(1) through 114(N)(referred to collectively as projected images 114). Image frames 102 maybe in any suitable video or still image format such as MPEG-2 (MovingPicture Experts Group), MPEG-4, JPEG (Joint Photographic Experts Group),JPEG 2000, TIFF (Tagged Image File Format), BMP (bit mapped format),RAW, PNG (Portable Network Graphics), GIF (Graphic Interchange Format),XPM (X Window System), SVG (Scalable Vector Graphics), and PPM (PortablePixel Map). Image display system 100 displays images 114 in at leastpartially overlapping positions (i.e., in a tiled format) on a displaysurface 116.

Image frame buffer 104 receives and buffers image frames 102. Framegenerator 108 processes buffered image frames 102 to form image frames110. In one embodiment, frame generator 108 processes a single stream ofimage frames 102 to form one or more image frames 110. In otherembodiments, frame generator 108 processes multiple streams of imageframes 102 to form one or more image frames 110.

Frame generator 108 processes image frames 102 to define image frames110(1) through 110(N) (collectively referred to as frames 110) usingrespective geometric meshes 126(1) through 126(N) (collectively referredto as geometric meshes 126) and respective photometric correctioninformation 128(1) through 128(N) (collectively referred to asphotometric correction information 128). Frame generator 108 providesframes 110(1) through 110(N) to projectors 112(1) through 112(N),respectively.

Projectors 112(1) through 112(N) store frames 110(1) through 110(N) inimage frame buffers 113(1) through 113 (N) (collectively referred to asimage frame buffers 113), respectively. Projectors 112(1) through 112(N)project frames 110(1) through 110(N), respectively, onto display surface116 to produce projected images 114(1) through 114(N) for viewing by oneor more users. Projectors 112 project frames 110 such that eachdisplayed image 114 at least partially overlaps with another displayedimage 114.

Projected images 114 are defined to include any combination ofpictorial, graphical, or textural characters, symbols, illustrations, orother representations of information. Projected images 114 may be stillimages, video images, or any combination of still and video images.

Display surface 116 includes any suitable surface configured to displayimages 114. In one or more embodiments described herein, display surface116 forms a developable surface. As used herein, the term developablesurface is defined as a surface that is formed by folding, bending,cutting, and otherwise manipulating a planar sheet of material withoutstretching the sheet. A developable surface may be planar, piecewiseplanar, or non-planar. A developable surface may form a shape such as acylindrical section or a parabolic section. As described in additionaldetail below, image display system 100 is configured to displayprojected images 114 onto a developable surface without geometricdistortion.

By displaying images 114 onto a developable surface, images 114 areprojected to appear as if they have been “wallpapered” to thedevelopable surface where no pixels of images 114 are stretched. Thewallpaper-like appearance of images 114 on a developable surface appearsto a viewer to be undistorted.

A developable surface can be described by the motion of a straight linesegment through three-dimensional (3D) space. FIG. 1B is a schematicdiagram illustrating a planar surface 130. As shown in FIG. 1B, planarsurface 130 is a shape that can be created by moving a straight linesegment λ through 3D space. E₁(t₁) and E₂(t₂) represent endpoint curves132 and 134 traced by the movement of the endpoints of the line segmentλ. Endpoint curves 132 and 134 swept out in 3D space by the endpoints ofthe line segment λ are sufficient to define the entire surface 130. Withplanar developable surface 130, endpoint curves 132 and 134 arestraight, parallel lines.

When planar surface 130 is curved into a non-planar developable surface140 without stretching as indicated by an arrow 136, the straightendpoint curves 132 and 134 become curved endpoint curves 142 and 144 inthe example of FIG. 1B. Curving planar surface 130 into non-planarsurface 140 may be thought of as analogous to bending, folding, orwallpapering planar surface 130 onto a curved surface withoutstretching. Endpoint curves 142 and 144 swept out in 3D space by theendpoints of the line segment λ are sufficient to define the entiresurface 140.

Image display system 100 may be configured to construct atwo-dimensional (2D) coordinate system corresponding to planar surface130 from which non-planar surface 140 was created using a predeterminedarrangement of identifiable points in fiducial marks 118 on displaysurface 116. The geometry of the predetermined arrangement ofidentifiable points may be described according to distance measurementsbetween the identifiable points. The distances between a predeterminedarrangement of points may all be scaled by a single scale factor withoutaffecting the relative geometry of the points, and hence the scale ofthe distances between the points on display surface 116 does not need tobe measured. In the embodiment shown in FIG. 1B, the predeterminedarrangement of points lie in fiducial marks 118 along the curvedendpoint curves E₁(t₁) and E₂(t₂) in display surface 116. These endpointcurves define a 2D coordinate system in the planar surface 130 createdby flattening curved display surface 140. Specifically, E₁(t₁) andE₂(t₂) are parallel in surface 130, with the connecting line segment λlying in the orthogonal direction at each t.

Non-planar developable display surfaces may allow a viewer to feelimmersed in the projected scene. In addition, such surfaces may fillmost or all of a viewer's field of view which allows scenes to be viewedas if they are at the same scale as they would be seen in the realworld.

Image display system 100 attempts to display images 114 on displaysurface 116 with a minimum amount of distortion, smooth brightnesslevels, and a smooth color gamut. To do so, frame generator 108 appliesgeometric and photometric correction to image frames 102 using geometricmeshes 126 and photometric correction information 128, respectively, inthe process of rendering frames 110. Geometric correction is describedin additional detail in Section II below, and photometric correction isdescribed in additional detail in Section III below.

Frame generator 108 may perform any suitable image decompression, colorprocessing, and conversion on image frames 102. For example, framegenerator 108 may convert image frames 102 from the YUV-4:2:0 format ofan MPEG2 video stream to an RGB format. In addition, frame generator 108may transform image frames 102 using a matrix multiply to translate,rotate, or scale image frames 102 prior to rendering. Frame generator108 may perform any image decompression, color processing, colorconversion, or image transforms prior to rendering image frames 102 withgeometric meshes 126 and photometric correction information 128.

Calibration unit 124 generates geometric meshes 126 and photometriccorrection information 128 using images 123 captured by at least onecamera 122 during a calibration process. Camera 122 may be any suitableimage capture device configured to capture images 123 of display surface116. Camera 122 captures images 123 such that the images includefiducial marks 118 (shown as fiducial marker strips 118A and 118B inFIGS. 1C and 1D) on display surface 116. Fiducial marks 118 may be anysuitable pattern or set of patterns that include a set of points withpredetermined arrangement of the points where the patterns arerecognizable by a pattern recognition algorithm. Fiducial marks 118 maybe permanently attached to display surface 116 or may be applied todisplay surface 116 only during the calibration process. Calibrationunit 124 uses the predetermined arrangement of points to create amapping of display surface 116. The predetermined arrangement ofidentifiable points may be described by distance measurements betweenthe identifiable points in the 2D space of flattened display surface116, where the scale of the distance measurements is not necessarilyknown. Fiducial marks 118 may be located outside of the display area ondisplay surface 116 where images 114 will appear when displayed byprojectors 112. In the embodiment shown in FIGS. 1C and 1D, fiducialmarker strips 118A and 118B form a black and white checkerboard patternsat the top and bottom of display surface 116 where the distance betweenthe corners of the checkerboard patterns in the horizontal direction isknown by image display system 10. In other embodiments, fiducial marks118 may form any other suitable pattern. In further embodiments,fiducial marks 118 may also consist of active light emitters, such asLEDs, lasers, or infrared light sources. These light sources mayoptionally be deactivated during display of images 114 on displaysurface 116.

In one embodiment, camera 122 includes a single camera configured tocapture image 123 that include the entirety of display surface 116. Inother embodiments, camera 122 includes multiple cameras each configuredto capture images 123 that include a portion of display surface 116where the combined images 123 of the multiple cameras include theentirety of display surface 116.

FIG. 1C is a schematic diagram illustrating the projection of partiallyoverlapping images 114(1) through 114(6) onto a non-planar developabledisplay surface 116 without correction. In FIG. 1B, images 114(1)through 114(6) appear as a set of distorted (i.e., warped) anddisjointed (i.e., unaligned) images. Each image 114(1) through 114(6)appears distorted because of the display of a planar image onto anon-planar surface, and the set of images 114(1) through 114(6) appearsdisjointed because images 114 are not spatially aligned or otherwisedisplayed in a uniform way on display surface 116.

Without photometric correction, regions of overlap between images 114may appear brighter than non-overlapping regions. In addition,variations between projectors 112 may result in variations in brightnessand color gamut between projected images 114(1) through 114(6).

FIG. 1D is a schematic diagram illustrating the projection of images114(1) through 114(6) onto non-planar developable display surface 116with geometric and photometric correction. By applying geometriccorrection as described in Section II below, frame generator 108unwarps, spatially aligns, and crops images 114(1) through 114(6) tominimize distortion in the display of images 114(1) through 114(6) ondisplay surface 116. Frame generator 108 also spatially aligns images114(1) through 114(6) as shown in FIG. 1D.

In addition, frame generator 108 may smooth any variations in brightnessand color gamut between projected images 114(1) through 114(6) byapplying photometric correction as described in Section III below. Forexample, frame generator 108 may smooth variations in brightness inoverlapping regions such as an overlapping region 150 between images114(1) and 114(2), an overlapping region 152 between images 114(2),114(3), and 114(4), and an overlapping region 154 between images 114(3),114(4), 114(5), and 114(6). Frame generator 108 may smooth variations inbrightness between images 114 displayed with different projectors 112.

Processing system 101 includes hardware, software, firmware, or acombination of these. In one embodiment, one or more components ofprocessing system 101 are included in a computer, computer server, orother microprocessor-based system capable of performing a sequence oflogic operations. In addition, processing can be distributed throughoutthe system with individual portions being implemented in separate systemcomponents, such as in a networked or multiple computing unitenvironment.

Image frame buffer 104 includes memory for storing one or more imageframes of the streams of image frames 102 for one or more image frames110. Thus, image frame buffer 104 constitutes a database of one or moreimage frames 102. Image frame buffers 113 also include memory forstoring frames 110. Although shown as separate frame buffers 113 inprojectors 112 in the embodiment of FIG. 1, frame buffers 113 may becombined (e.g., into a single frame buffer) and may be external toprojectors 112 (e.g., in processing system 101 or between processingsystem 101 and projectors 112) in other embodiments. Examples of imageframe buffers 104 and 113 include non-volatile memory (e.g., a hard diskdrive or other persistent storage device) and volatile memory (e.g.,random access memory (RAM)).

It will be understood by a person of ordinary skill in the art thatfunctions performed by processing system 101, including frame generator108 and calibration unit 124, may be implemented in hardware, software,firmware, or any combination thereof. The implementation may be via oneor more microprocessors, graphics processing units (GPUs), programmablelogic devices, or state machines. In addition, functions of framegenerator 108 and calibration unit 124 may be performed by separateprocessing systems in other embodiments. In such embodiments, geometricmeshes 126 and photometric correction information 128 may be providedfrom calibration unit 124 to frame generator 108 using any suitablewired or wireless connection or any suitable intermediate storagedevice. Components of the present invention may reside in software onone or more computer-readable mediums. The term computer-readable mediumas used herein is defined to include any kind of memory, volatile ornon-volatile, such as floppy disks, hard disks, CD-ROMs, flash memory,read-only memory, and random access memory.

II. Geometric Calibration and Correction of Displayed Images

Image display system 100 applies geometric correction to image frames102 as part of the process of rendering image frames 110. As a result ofthe geometric correction, image display system 100 displays images 114on display surface 116 using image frames 110 such that viewers may viewimages as being undistorted for all viewpoints of display surface 116.

Image display system 100 generates geometric meshes 126 as part of ageometric calibration process. Image display system 100 determinesgeometric meshes 126 using predetermined arrangements between points offiducial marks 118. Image display system 100 determines geometric meshes126 without knowing the shape or any dimensions of display surface 116other than the predetermined arrangements of points of fiducial marks118.

Frame generator 108 renders image frames 110 using respective geometricmeshes 126 to unwarp, spatially align, and crop frames 102 into shapesthat are suitable for display on display surface 116. Frame generator108 renders image frames 110 to create precise pixel alignment betweenoverlapping images 114 in the overlap regions (e.g., regions 150, 152,and 152 in FIG. 1D).

In the following description of generating and using geometric meshes126, four types of 2D coordinate systems will be discussed. First, aprojector domain coordinate system, P_(i), represents coordinates inframe buffer 113 of the ith projector 112. Second, a camera domaincoordinate system, C_(j), represents coordinates in images 123 capturedby the jth camera 122. Third, a screen domain coordinate system, S,represents coordinates in the plane formed by flattening display surface116. Fourth, an image frame domain coordinate system, I, representcoordinates within image frames 102 to be rendered by frame generator108.

Image display system 100 performs geometric correction on image frames102 to conform images 114 from image frames 102 to display surface 116without distortion. Accordingly, in the case of a single input imagestream, the image frame domain coordinate system, I, of image frames 102may be considered equivalent to the screen domain coordinate system, S,up to a scale in each of the two dimensions. By normalizing bothcoordinate systems to the range [0, 1], the image frame domaincoordinate system, I, becomes identical to the screen domain coordinatesystem, S. Therefore, if mappings between the screen domain coordinatesystem, S, and each projector domain coordinate system, P_(i), aredetermined, then the mappings from each projector domain coordinatesystem, P_(i), to the image frame domain coordinate system, I, maydetermined.

Let P_(i)({right arrow over (s)}) be a continuous-valued function thatmaps 2D screen coordinates {right arrow over (s)}=(s_(x),s_(y)) in S tocoordinates {right arrow over (p)}=(p_(x,i), p_(y,i)) of the ithprojector 112's frame buffer 113. P_(i) is constructed as a compositionof two coordinate mappings as shown in Equation 1:

{right arrow over (p)} _(i) =P _(i)({right arrow over (s)})=C _(i,j)(S_(j)({right arrow over (s)}))   (1)

where S_(j)({right arrow over (s)}) is a 2D mapping from display surface116 to the image pixel locations of the jth observing camera 122, andC_(i,j)({right arrow over (c)}_(j)) is a 2D mapping from image pixellocations {right arrow over (c)}=(c_(x,j),c_(y,j)) of the jth observingcamera 122 to the ith projector 112's frame buffer 113. If all S_(j) andC_(ij) are invertible mappings, the mappings from projector framebuffers to the flattened screen are constructed similarly from theinverses of the S_(j) and C_(ij) mappings, as shown in Equation 2:

{right arrow over (s)}=P _(i) ⁻¹({right arrow over (p)} _(i))=S _(j)⁻¹(C _(i,j) ⁻¹({right arrow over (p)} _(i)))   (2)

Hence, all coordinate transforms required by the geometric correctioncan be derived from the S_(j) and C_(ij) mappings.

To handle a broad set of screen shapes, image display system 100constructs generalized, non-parametric forms of these coordinatemappings. Specifically, for each mapping, image display system 100 usesa mesh-based coordinate transform derived from a set of pointcorrespondences between the coordinate systems of interest.

Given a set of point correspondences between two 2D domains A and B,image display system 100 maps a point location {right arrow over (a)} inA to a coordinate {right arrow over (b)} in B as follows. Image displaysystem 100 applies Delaunay triangulation to the points in A to create afirst triangle mesh and then constructs the corresponding triangle mesh(according to the set of point correspondences) in B. To determine apoint {right arrow over (b)} that corresponds to a point {right arrowover (a)}, image display system 100 finds the triangle in the trianglemesh in domain A that contains {right arrow over (a)}, or whose centroidis closest to it, and computes the barycentric coordinates of {rightarrow over (a)} with respect to that triangle. Image display system 100then selects the corresponding triangle from the triangle mesh in domainB and computes {right arrow over (b)} as the point having these samebarycentric coordinates with respect to the triangle in B. Image displaysystem 100 determines a point {right arrow over (a)} that corresponds toa point {right arrow over (b)} similarly.

The geometric meshes used to perform coordinate mappings have theadvantage of allowing construction of coordinate mappings from pointcorrespondences where the points in either domain may be in anyarrangement other than collinear. This in turn allows greaterflexibility in the calibration methods used for measuring the locationsof the points involved in the point correspondences. For example, thepoints on display surface 116 may be located entirely outside the areaused to display projected images 114, so that these points do notinterfere with displayed imagery, and may be left in place while thedisplay is in use. Other non-parametric representations of coordinatemappings, such as 2D lookup tables, are generally constructed from 2Darrays of point correspondences. In many instances it is not convenientto use 2D arrays of points. For example, a 2D array of points on displaysurface 116 may interfere with displayed imagery 114, so that thesepoints may need to be removed after calibration and prior to use of thedisplay. Also, meshes may more easily allow for spatial variation in thefineness of the coordinate mappings, so that more point correspondencesand triangles may be used in display surface areas that require finercalibration. Finer mesh detail may be localized independently tospecific 2D regions within meshes by using more point correspondences inthese regions, whereas increased fineness in the rows or columns of a 2Dlookup table generally affects a coordinate mapping across the entirewidth or height extent of the mapping. In many instances, a mesh-basedrepresentation of a coordinate mapping may also be more compact, andhence require less storage and less computation during the mappingprocess, than a similarly accurate coordinate mapping stored in anothernon-parametric form such as a lookup table.

To determine the correct projector frame buffer contents needed torender the input image like wallpaper on the screen, image displaysystem 100 applies Equation 2 to determine the screen location {rightarrow over (s)} that each projector pixel {right arrow over (p)} lightsup. If {right arrow over (s)} is normalized to [0, 1] in bothdimensions, then this is also the coordinate for the input image pixelwhose color should be placed in {right arrow over (p)}, sincewallpapering the screen effectively equates the 2D flattened screencoordinate systems S with the image coordinate system I. For eachprojector 112, image display system 100 uses Equation 2 to compute theimage coordinates corresponding to each location on a sparsely sampledrectangular grid (e.g., a 20×20 grid) in the screen coordinate space.Graphics hardware fills the projector frame buffer via texture mappingimage interpolation. Hence, the final output of the geometriccalibration is one triangle mesh 126 per projector 112, computed on therectangular grid.

Because the method just described includes a dense mapping to thephysical screen coordinate system, it corrects for image distortioncaused not only by screen curvature, but also due to the projectorlenses. Furthermore, the lens distortion of the observing camera(s) 122,inserted by interposing their coordinate systems between those of theprojectors and the screen, does not need to be calibrated and corrected.In fact, the method allows use of cameras 122 with extremely wide anglelenses, without any need for camera image undistortion. Because of this,image display system 100 may be calibrated with a single, wide-anglecamera 122. This approach can even be used to calibrate full 360 degreedisplays, by placing a conical mirror in front of the camera lens toobtain a panoramic field-of-view.

Methods of performing geometric correction will now be described inadditional detail with reference to the embodiments of FIGS. 2A-2H.FIGS. 2A-2H are flow charts illustrating methods for geometriccorrection. FIG. 2A illustrates the overall calibration process togenerate geometric meshes 126, and FIG. 2B illustrates the renderingprocess using geometric meshes 126 to perform geometric correction onimage frames 102. FIGS. 2C through 2H illustrate additional details ofthe functions of the blocks shown in FIGS. 2A and 2B. The embodiments ofFIGS. 2A-2H will be described with reference to image display system 100as illustrated in FIG. 1.

The methods of FIGS. 2A-2H will be described for an embodiment of imagedisplay system 100 that includes a single camera 122. In embodimentsthat include multiple cameras 122, then methods of FIGS. 2A-2H may begeneralized for multiple cameras 122 using Equations 1 and 2 above. Withmultiple cameras 122, image display system 100 may also align meshesfrom multiple cameras 122 onto a single mesh in the camera domain. Whenfields-of-view of multiple cameras overlap the same screen or projectorregion, mesh-based coordinate mapping results from different cameras 122may be combined in a weighted average, with the weights optionally beingdetermined by the distance of the location from the edges of the camerafields-of-view. In addition, image display system 100 registers thedifferent camera coordinate systems using projector or screen pointsfrom their overlap regions, and/or using any of the many methods formulti-camera geometric calibration known in art.

In the embodiments described below, geometric meshes 126 will bedescribed as triangle meshes where each triangle mesh forms a set oftriangles where each triangle is described with a set of threecoordinate locations (i.e., vertices). Each triangle in a triangle meshcorresponds to another triangle (i.e., a set of three coordinatelocations or vertices) in another triangle mesh from another domain.Accordingly, corresponding triangles in two domains may be representedby six coordinate locations—three coordinate locations in the firstdomain and three coordinate locations in the second domain.

In other embodiments, geometric meshes 126 may be polygonal meshes withpolygons with z sides, where z is greater than or equal to four. Inthese embodiments, corresponding polygons in two domains may berepresented by 2z ordered coordinate locations—z ordered coordinatelocations in the first domain and z ordered coordinate locations in thesecond domain.

In FIG. 2A, calibration unit 124 generates screen-to-camera trianglemeshes as indicated in a block 202. In particular, calibration unit 124generates a triangle mesh in the screen domain and a correspondingtriangle mesh in the camera domain. Calibration unit 124 generates thesetriangle meshes using knowledge of a predetermined arrangement offiducial marks 118, and an image 123 captured by camera 122 thatincludes these fiducial marks 118 on display surface 116.

Calibration unit 124 also generates camera-to-projector triangle meshesfor each projector 112 as indicated in a block 204. In particular,calibration unit 124 generates a second triangle mesh in the cameradomain and a corresponding triangle mesh in the projector domain foreach projector 112. Calibration unit 124 generates these triangle meshesfrom known pattern sequences displayed by projectors 112 and a set ofimages 123 captured by camera 122 viewing display surface 116 whilethese known pattern sequences are projected by projectors 112.

Calibration unit 124 generates a screen-to-projector triangle mesh, alsoreferred to as geometric mesh 126, for each projector 112 as indicatedin a block 206. Calibration unit 124 generates geometric meshes 126 suchthat each geometric mesh 126 includes a set of points that areassociated with a respective projector 112. Calibration unit 124identifies the set of points for each projector 112 using thescreen-to-camera triangle meshes and the camera-to-projector trianglemeshes as described in additional detail below with reference to FIGS.2F and 2G.

Referring to FIG. 2B, frame generator 108 renders frames 110 for eachprojector 112 using the respective geometric mesh 126 as indicated in ablock 208. Frame generator 108 provides respective frames 110 torespective frame buffers 113 in respective projectors 112. Projectors112 project respective frames 110 onto display surface 116 in partiallyoverlapping positions as indicated in a block 210. Because eachgeometric mesh 126 defines a mapping between display surface 116 and aframe buffer 113 of a respective projector 112, frame generator 108 usesgeometric meshes 126 to warp frames 102 into frames 110 such that frames110 appear spatially aligned and without distortion when projected byprojectors 112 as images 114 in partially overlapping positions ondisplay surface 116. Frame generator 108 interpolates the pixel valuesfor frames 110 using the geometric meshes 126 as described in additionaldetail below with reference to FIG. 2H.

FIG. 2C illustrates a method for performing the function of block 202 ofFIG. 2A. Namely, the method of FIG. 2C illustrates one embodiment ofgenerating screen-to-camera triangle meshes. The method of FIG. 2C willbe described with reference to FIGS. 3A-3D.

In FIG. 2C, camera 122 captures an image 123A (shown in FIG. 3A) ofdisplay surface 116 that includes fiducial marks 118 as indicated in ablock 212. Fiducial marks 118 include points identifiable in image 123Aby calibration unit 124 where the arrangement of the points ispredetermined. For example, fiducial marks 118 may form a black andwhite checkerboard pattern where the distances between all adjacentcorners are the same linear distance.

Calibration unit 124 locates fiducial marks 118 in image 123A asindicated in a block 214. Calibration unit 124 locates fiducial marks118 to identify points where points are located according to apredetermined arrangement on display screen 116. For example, wherefiducial marks 118 form a black and white checkerboard pattern as in theexample shown in FIG. 1D, calibration unit 124 may detect the pointsusing a standard corner detector along with the following algorithm suchthat the detected corners form the points located according to apredetermined arrangement on display screen 116.

In one embodiment, calibration unit 124 assumes the center of image 123Ais inside the region of display surface 116 to be used for display,where this region is at least partially bounded by strips of fiducialsmarks 118, and where the region contains no fiducial marks 118 in itsinterior. The boundary of the region along which fiducial marks 118appear may coincide with the boundary of display surface 116, or mayfall entirely or partially in the interior of display surface 116. FIG.1C shows example strips 118A and 118B located along the top and bottomborders of display surface 116. The strips contain checkerboardpatterns, with all squares having equal size. The physical size of thesesquares is predetermined, and therefore the physical distances along thescreen surface between successive corners on the interior horizontalline within each strip is known.

Calibration unit 124 begins searching from the center of camera image123A going upward for the lowest detected corner. Referring back tofiducial marker strip 118A in FIG. 1D, calibration unit 124 may assumethat this lowest detected corner (i.e., the first fiducial mark) is onthe bottom row of fiducial marker strip 118A. Calibration unit 124 findsthe next lowest corner searching upward (e.g., an interior corner of thecheckerboard pattern) and saves the vertical distance from the firstcorner to the next lowest corner as a vertical pattern step.

Calibration unit 124 searches left from the interior corner forsuccessive corners along fiducial marker strip 118A at the step distance(estimating the horizontal pattern step to be equal to the verticalpattern step), plus or minus a tolerance, until no more corners aredetected in the expected locations. In traversing the image of the stripof fiducial marker strip 118A, calibration unit 124 predicts thelocation of the next corner in sequence by extrapolating using thepattern step to estimate the 2D displacement in camera image 123A fromthe previous corner to the next corner. By doing so, calibration unit124 may follow accurately the smooth curve of the upper strip offiducial marks 118 which appears in image 123A.

Calibration unit 124 then returns to the first fiducial location andcontinues the search to the right in a manner analogous to thatdescribed for searching to the left. Calibration unit 124 subsequentlyreturns to the center of camera image 123A, and searches downward tolocate a first corner in fiducial marks 118B. This corner is assumed tobe on the top row of fiducial marker strip 118B. The procedure used forfinding all corners in upper fiducial strip 118A is then carried out inan analogous way for the lower strip, this time using the corners in therow of fiducial strip 118B below the row containing the first detectedcorner. Searches to the left and right are carried out as before, andlocations of all corners in the middle row of fiducial strip 118B arestored.

In FIG. 3A, points 300 represent the points in a screen domain (S) 302that are separated by an example predetermined arrangement—with apredetermined separation distance (d1) in the horizontal direction and apredetermined separation distance (d2) in the vertical direction ondisplay screen 116. Points 310 represent the points in a camera domain(C) 312 that are identified in image 123A by calibration unit 124 asjust described (e.g., as interior corner locations of a black and whitecheckerboard pattern). In other embodiments, points 300 may be arrangedwith other known geometry, distances, and/or other scaling informationbetween points 300.

Referring to FIGS. 2C and 3A, calibration unit 124 generates a set ofpoint correspondences 308 between fiducial marks 118 detected in image123A and fiducial marks 118 on display surface 116 as indicated in ablock 216. The set of point correspondences 308 are represented byarrows that identify corresponding points in screen domain 302 andcamera domain 312. These correspondences are generated by matchingdetected fiducials marks in camera image 123A with the predeterminedarrangement of fiducial marks 118 on display surface 116. The algorithmdescribed above for fiducial strips 118A and 118B describes one methodfor making these correspondences for a particular arrangement offiducial marks 118, but other algorithms can be used for otherarrangements of fiducial marks.

Calibration unit 124 determines screen-to-camera triangle meshes usingthe set of correspondences 308 as indicated in a block 218. Thescreen-to-camera triangle meshes are used to map screen domain (S) 302to camera domain (C) 312 and vice versa. Calibration unit 124 determinesscreen-to-camera triangle meshes using the method illustrated in FIG.2D. FIG. 2D illustrates a method for generating a triangle mesh in eachof two domains.

Referring to FIG. 2D and FIG. 3B, calibration unit 124 constructs afirst triangle mesh in a first domain as indicated in a block 222. Inthe example of FIG. 3B, calibration unit 124 constructs a triangle mesh304 in screen domain 302 by connecting points 300. Calibration unit 124constructs triangle mesh 304 using Delaunay triangulation or any othersuitable triangulation algorithm.

Calibration unit 124 constructs a second triangle mesh in a seconddomain that corresponds to the first triangle mesh using a set of pointcorrespondences as indicated in a block 224. Referring to FIG. 3C,calibration unit 124 constructs a triangle mesh 314 in camera domain 312by connecting points 310 in the same way that corresponding points 300,according to point correspondences 308, are connected in screen domain302.

Calibration unit 124 uses the set of point correspondences 308 to ensurethat triangles in triangle mesh 314 correspond to triangles in trianglemesh 304. For example, points 300A, 300B, and 300C correspond to points310A, 310B, and 310C as shown by the set of point correspondences 308.Accordingly, because calibration unit 124 formed a triangle 304A intriangle mesh 304 using points 300A, 300B, and 300C, calibration unit124 also forms a triangle 314A in triangle mesh 314 using points 310A,310B, and 310C. Triangle 314A therefore corresponds to triangle 304A.

In other embodiments, calibration unit 124 may first a constructtriangle mesh 314 in camera domain 312 (e.g. by Delaunay triangulation)and then construct triangle mesh 304 in screen domain 302 using the setof point correspondences 308.

FIG. 2E illustrates a method for performing the function of block 204 ofFIG. 2A. Namely, the method of FIG. 2E illustrates one embodiment ofgenerating camera-to-projector triangle meshes. The method of FIG. 2Ewill be described with reference to FIGS. 4A-4D. The method of FIG. 2Eis performed for each projector 112 to generate camera-to-projectortriangle meshes for each projector 112.

In FIG. 2E, calibration unit 124 causes a projector 112 to display a setof known pattern sequences on display surface 116 as indicated in ablock 220. Calibration unit 124 provides a series of frames 110 withknown patterns to frame buffer 113 in projector 112 by way of framegenerator 108. Projector 112 displays the series of known patterns.

Camera 122 captures a set of images 123B (shown in FIG. 4A) of displaysurface 116 while the known patterns are being projected onto displaysurface 116 by projector 112 as indicated in a block 232. The knownpatterns may be any suitable patterns that allow calibration unit 124 toidentify points in the patterns using images 123B captured by camera122. For example, the known patterns may be a sequence of horizontal andvertical black-and-white bar patterns.

Calibration unit 124 locates points of the known patterns in images 123Bas indicated in a block 234. In FIG. 4A, points 400 represent the pointsin camera domain (C) 312 located by calibration unit 124. In oneembodiment, calibration unit 124 locates the points by projecting aknown series of known black-and-white patterns onto display surface 116,and then correlating sequences of black and white pixel observations inimages 123B of these known patterns with the sequences of black andwhite values at locations within the projected pattern coordinate space.For each camera image 123B of a known pattern, pixels are classified ascorresponding to a black projected pattern element, a white projectedpattern element, or being outside the coverage area of the projector.Each camera pixel location within the coverage area of the projector isthen assigned a black/white bit-sequence summarizing the sequence ofobservations found while the known patterns were displayed in sequence.Calibration unit 124 uses the bit sequences as position codes for thecamera pixels. A camera location image may be formed to display theposition codes for each camera pixel. The camera location image may bedivided into code set regions, each region containing camera pixellocations all having an identical associated black/white bit sequence.The size and number of code set regions in the camera location imagedepends upon the number and fineness of the bar patterns. A similarprojector location image may be formed by displaying the black/white bitsequences at each projector pixel location as the known patterns werebeing displayed in a known sequence. The projector location image mayalso be divided into position code set regions, each region containingprojector pixels all having an identical associated black/white bitsequence. A correspondence between code set regions in the camera andprojector location images is made by matching the black/white bitsequence position codes of respective regions in the two images.Calibration unit 124 computes the centers-of-mass of the detected codeset regions in the camera location image as the points to be associatedwith the centers-of-mass of the corresponding code set regions in theprojector location image of projector 112.

Referring to FIGS. 2E and 4A, calibration unit 124 generates a set ofpoint correspondences 308 between the known patterns (in the coordinatespace of projector 112) and camera images 123B of these known patternsas indicated in a block 236. Points 410(i) represent the ith points(where i is between 1 and N) in an ith projector domain (P_(i)) 412(i)that are identified in image 123B by calibration unit 124. The ith setof point correspondences 408(i) are represented by arrows that identifycorresponding points in camera domain 312 and projector domain 412(i).

In one embodiment, calibration unit 124 associates the centers-of-massof the detected position code sets in the camera location image (i.e.,points 400) with the centers-of-mass of the corresponding position codesets (i.e., points 410(i) of the known patterns) provided toframe-buffer 113 of projector 112 to generate the set of pointcorrespondences 308.

Calibration unit 124 determines camera-to-projector triangle meshesusing the set of correspondences 408(i) as indicated in a block 238. Thecamera-to-projector triangle meshes are used to map camera domain (C)312 to projector domain (P_(i)) 412(i) and vice versa. Calibration unit124 determines camera-to-projector triangle meshes using the methodillustrated in FIG. 2D.

Referring to FIG. 2D and FIG. 4B, calibration unit 124 constructs afirst triangle mesh in a first domain as indicated in block 222. In theexample of FIG. 4B, calibration unit 124 constructs a triangle mesh 404in camera domain 312 by connecting points 400. Calibration unit 124constructs triangle mesh 404 using Delaunay triangulation or any othersuitable triangulation algorithm.

Calibration unit 124 constructs a second triangle mesh in a seconddomain that corresponds to the first triangle mesh using a set of pointcorrespondences as indicated in block 224. Referring to FIG. 4C,calibration unit 124 constructs a triangle mesh 414(i) in projectordomain 412(i) by connecting points 410(i) using the set of pointcorrespondences 408(i) in the same way that corresponding points 400,according to point correspondences 408(i), are connected in cameradomain 312.

Calibration unit 124 uses the set of point correspondences 408(i) toensure that triangles in triangle mesh 414(i) correspond to triangles intriangle mesh 404. For example, points 400A, 400B, and 400C correspondto points 410(i)A, 410(i)B, and 410(i)C as shown by the set of pointcorrespondences 408(i). Accordingly, because calibration unit 124 formeda triangle 404A in triangle mesh 404 using points 400A, 400B, and 400C,calibration unit 124 also forms a triangle 414(i)A in triangle mesh414(i) using points 410(i)A, 410(i)B, and 410(i)C. Triangle 414(i)Atherefore corresponds to triangle 404A.

In other embodiments, calibration unit 124 may first construct trianglemesh 414(i) in projector domain 412(i) and then construct triangle mesh404 in camera domain 312 using the set of point correspondences 408(i).

Referring back to block 206 of FIG. 2A, calibration unit 124 generates ageometric mesh 126 for each projector 112 using the screen-to-camerameshes (block 202 and FIG. 2C) and camera-to-projector meshes for eachprojector 112 (block 204 and FIG. 2E). Each geometric mesh 126 mapsscreen domain (S) 302 to a projector domain (P_(i)) 412 and vice versa.

FIG. 2F illustrates a method for performing the function of block 206 ofFIG. 2A. Namely, the method of FIG. 2F illustrates one embodiment ofgenerating a geometric mesh 126 that maps the screen domain to aprojector domain of a projector 112. The method of FIG. 2F will bedescribed with reference to the example of FIG. 5A. The method of FIG.2F is performed for each projector 112 to generate geometric meshes126(1) through 126(N) for respective projectors 112(1) through 112(N).

The method FIG. 2F will be described below for generating geometric mesh126(1). Geometric meshes 126(2) through 126(N) are generated similarly.

Referring to FIGS. 2F and 5A, calibration unit 124 constructs a trianglemesh 502 over a rectangular, evenly spaced grid that includes a set ofpoints 500 in screen domain 302 as indicated in a block 242. In otherembodiments, triangle mesh 502 may be constructed over arrangements ofpoints 500 other than rectangular, evenly-spaced grids. The set ofpoints 500 occur at least partially in a region 504(1) of screen domain302 where projector 112(1) is configured to display image 114(1).Delaunay triangulation or other suitable triangulation methods are usedto construct a triangle mesh from the set of points 500(1).

Calibration unit 124 generates a set of point correspondences 508(1)between the set of points 500 in screen domain 302 and a set of points510(1) in projector domain 412(1) using the screen-to-camera meshes andthe camera-to-projector meshes for projector 112(1) as indicated in ablock 244.

FIG. 2G illustrates one embodiment of a method for generating a pointcorrespondence in the set of point correspondences 508(1) in block 244of FIG. 2F. The method of FIG. 2G will be described with reference toFIGS. 3D and 4D.

In FIG. 2G, calibration unit 124 identifies a triangle in the screentriangle mesh (determined in block 218 of FIG. 2C) that includes or isnearest to a point in the screen domain as indicated in a block 252. InFIG. 3D, for example, calibration unit 124 identifies triangle 304A intriangle mesh 304 that includes a point 306 in screen domain 302.

Calibration unit 124 determines barycentric coordinates for the point inthe triangle in the screen domain as indicated in a block 254. In theexample of FIG. 3D, calibration unit 124 determines barycentriccoordinates for point 306 in triangle 304A, as represented by the dottedlines that connect point 306 to the vertices of triangle 304A, in screendomain 302.

Calibration unit 124 applies the barycentric coordinates to acorresponding triangle in the camera triangle mesh (determined in block218 of FIG. 2C) to identify a point in the camera domain thatcorresponds to the point in the screen domain as indicated in a block256. In the example of FIG. 3D, calibration unit 124 applies thebarycentric coordinates to a corresponding triangle 314A in trianglemesh 314 to identify a point 316 in camera domain 312 that correspondsto point 306 in screen domain 302.

Calibration unit 124 identifies a triangle in the camera triangle mesh(as determined in block 238 of FIG. 2E) that includes or is nearest tothe point in the camera domain as indicated in a block 258. In FIG. 4D,for example, calibration unit 124 identifies triangle 404A in trianglemesh 404 that includes point 316 in camera domain 312.

Calibration unit 124 determines barycentric coordinates for the point inthe triangle in the camera domain as indicated in a block 260. In theexample of FIG. 4D, calibration unit 124 determines barycentriccoordinates for point 316 in triangle 404A, as represented by the dottedlines that connect point 316 to the vertices of triangle 404A, in cameradomain 312.

Calibration unit 124 applies the barycentric coordinates to acorresponding triangle in the projector triangle mesh (as determined inblock 238 of FIG. 2E) to identify a point in the projector domain thatcorresponds to the point in the camera domain as indicated in a block262. In the example of FIG. 4D, calibration unit 124 applies thebarycentric coordinates to a corresponding triangle 414(i)A in trianglemesh 414(i) to identify a point 416 in projector domain 412(i) thatcorresponds to point 316 in screen domain 312.

By performing the method of FIG. 2G, calibration unit 124 generates apoint correspondence in the set of point correspondences 508(1). In theexample of FIGS. 3D and 4D, calibration unit 124 generates a pointcorrespondence between point 306 in screen domain 302 and point 416 inprojector domain 412(i) using screen-to-camera meshes 304 and 314 andcamera-to-projector meshes 404 and 414(i). The method of FIG. 2G isrepeated for each selected point of triangle mesh 502 to generate theremaining point correspondences in the set of point correspondences508(1).

Referring back to FIGS. 2F and 5A, calibration unit 124 constructs ageometric triangle mesh 126(1) in projector domain 412(1) thatcorresponds to triangle mesh 502 in screen domain 302 using the set ofpoint correspondences 508(1) as indicated in a block 246. Calibrationunit 124 constructs geometric triangle mesh 126(1) in projector domain412(1) by connecting points 510(1) according to the set of pointcorrespondences 508(1). Calibration unit 124 uses the set of pointcorrespondences 508(1) to ensure that triangles in triangle mesh 126(1)correspond to triangles in triangle mesh 502.

In other embodiments, calibration unit 124 may first construct trianglemesh 126(1) in projector domain 412(1), using Delaunay triangulation orother suitable triangulation methods, and then construct triangle mesh502 in screen domain 312 using the set of point correspondences 508(1).

Referring back to block 208 of FIG. 2B, frame generator 108 rendersframes 110 using respective geometric meshes 126. FIG. 2H illustrates amethod for mapping locations in frames 110 to locations in projectorframe buffers 113 to allow the function of block 208 to be performed.The method of FIG. 2H is performed by frame generator 108 for each pixelin each frame 110 using a respective geometric mesh 126 to determine thepixel colors of frame 110. The method of FIG. 2H will now be describedas being performed by frame generator 108 for a frame 110(1). Framegenerator 108 performs the method of FIG. 2H for frames 110(2) through110(N) similarly. The method of FIG. 2H will be described with referenceto an example in FIG. 5B.

Referring to FIG. 2H and 5B, frame generator 108 identifies a trianglein a respective projector triangle mesh that includes or is nearest to apixel in frame 110(1) as indicated in a block 272. The projectortriangle mesh, in the context of rendering, refers to a geometric mesh126(1) from block 246 of FIG. 2F that was constructed to correspond toscreen triangle mesh 502. In FIG. 5B, for example, frame generator 108identifies triangle 126(1)A in geometric mesh 126 that includes point520. A coordinate correspondence is also made between screen domain 302and the image domain I of an image frame 102 to be displayed. Thecorrespondence may include scaling, rotation, and translation, so that arectangular portion of image frame 102 may correspond to any rectangularregion of the 2D plane made by flattening display surface 116. Becauseof this coordinate correspondence between image domain I and screendomain 302, triangle mesh 502 corresponds to the image domain, I, offrame 102 as described in additional detail above.

Frame generator 108 determines barycentric coordinates for a pixellocation in frame buffer 113(1) in the triangle of projector trianglemesh 126(1) as indicated in a block 274. In the example of FIG. 5B,frame generator 108 determines barycentric coordinates for point 520 intriangle 126(1)A, as represented by the dotted lines that connect point520 to the vertices of triangle 126(1)A.

Frame generator 108 applies the barycentric coordinates to acorresponding triangle in screen triangle mesh 502 to identify a screenlocation, and hence a corresponding pixel location in image frame 102,as indicated in a block 276. In the example of FIG. 5B, frame generator108 applies the barycentric coordinates to a corresponding triangle 502Ain triangle mesh 502 to identify a point 522 in that corresponds topoint 520 in as indicated by a dashed arrow 526. Point 522 correspondsto a point 524 in image frame 102(1) in as indicated by a dashed arrow528. The color at this pixel location in frame buffer 113(1) is filledin with the color of the image data at the image domain I locationcorresponding to the screen location in screen triangle mesh 502.

Interpolation of image color between pixel locations in image domain Imay be used as part of this process, if the location determined in imageframe 102 is non-integral. This technique may be implemented efficientlyby using the texture mapping capabilities of many standard personalcomputer graphics hardware cards. In other embodiments, alternativetechniques for warping frames 102 to correct for geometric distortionusing geometric meshes 126 may be used, including forward mappingmethods that map from coordinates of image frames 102 to pixel locationin projector frame buffers 113 (via screen-to-projector mappings) toselect the pixel colors of image frames 102 to be drawn into projectorframe buffers 113.

By mapping frames 102 to projector frame buffers 113, frame generator108 may warp frames 102 into frames 110 to geometrically correct thedisplay of images 114.

Although the above methods contemplate the use of an embodiment ofdisplay system 100 with multiple projectors 112, the above methods mayalso be applied to an embodiment with a single projector 112.

In addition, the above method may be used to perform geometriccorrection on non-developable display surfaces. Doing so, however, mayresult in distortion that is visible to a viewer of the display surface.

III. Photometric Calibrations and Correction of Displayed Images

Even after geometric correction, the brightness of projected images 114is higher in screen regions of images 114 that overlap (e.g., regions150, 152, and 154 shown in FIG. 1D). In addition, light leakage in eachprojector 112 may cause a non-zero “black offset” to be projected ondisplay surface 116 for black image inputs. These black offsets have thepotential to add up in overlap regions to produce visually disturbingartifacts. Further, projector tone reproduction functions (TRFs) thatrelate output light color to image input values may vary acrossprojectors 112, as well as across pixels within a single projector 112,so that noticeable color and brightness transitions appear in thedisplay of images 114. For example, maximum projector brightness maydecrease toward the edge of the frustrum of a projector 112.

Image display system 100 applies photometric correction to image frames102 using photometric correction information 128 in the process ofrendering image frames 110 to cause smooth brightness levels and colorgamut across the combination of projected images 114 on display surface116. Accordingly, image display system 100 attempts to produce a tileddisplay system that will not produce visually disturbing colorvariations in a displayed image 114 for an input image frame 102 of anysingle solid color. By doing so, image display system 100 may implementphotometric correction while ensuring that projected images 114 appearreasonably faithful to the images of image frames 102.

Processing system 101 applies photometric correction by linearizing,scaling, and offsetting geometrically corrected frames 110A (shown inFIG. 7) to generate photometrically corrected frames 110B (shown in FIG.7) in one embodiment. Processing system 101 adds a black offset image(e.g., an offset map 704 shown in FIG. 7) to each frame 110A in order tocreate a smooth black level across images 114. Processing system 101applies a multiplicative attenuation (scaling) map (e.g., a scale map706 shown in FIG. 7) to pixel values in each frame 110A in order tosmooth the spatial variation of the brightnesses across images 114.Processing system 101 also applies a blend map (e.g., a blend map 702shown in FIG. 7) to each frame 110A for attenuating regions of displaysurface 116 where images 114 overlap. The blend maps spatially“cross-fade” the brightness levels of respective projectors 112 inoverlap regions between two or more projectors. Processing system 101linearizes the TRFs of projectors 112 to allow the same attenuation mapsto be used for all inputs. To do so, processing system 101 appliesinverse TRFs to frames 110A prior to providing image frames 110B toprojectors 112. The combination of this inversion and the physicalprojectors 112 may be considered together as linear projectors 112.Processing system 101 also applies a gamma function to frames 110A toprevent images 114 from appearing saturated as a result of replacingwith a linear pass-through the standard nonlinear “gamma” exponentialfunction typically applied to images.

Methods of performing photometric calibration and correction will now bedescribed in additional detail with reference to the embodiments ofFIGS. 6A-6G. FIGS. 6A-6G are flow charts illustrating methods forphotometric calibration and correction. FIG. 6A illustrates the overallcalibration process to generate photometric correction information 128,and FIG. 6B illustrates the rendering process using photometriccorrection information 128 to perform photometric correction on imageframes 110A. FIGS. 6C through 6G illustrate additional details of thefunctions of the blocks shown in FIGS. 6A. The embodiments of FIGS.6A-6G will be described with reference to image display system 100 asillustrated in FIG. 1A.

The methods of FIGS. 6A-6G will be described for an embodiment of imagedisplay system 100 that includes a single camera 122. In embodimentsthat include multiple cameras 122, the methods of FIGS. 6A-6G may beperformed using multiple cameras 122 by synchronizing and determiningthe geometric relationship between images 123 captured by cameras 122prior to performing the functions of methods of FIGS. 6C, 6D, 6F, and6G. Determination of the geometric relationship between images 123captured by different cameras 122 may be accomplished by any suitablemulti-camera geometric calibration method.

In FIG. 6A, calibration unit 124 causes projectors 112 to project aseries of gray levels onto display surface 116 and camera 112 capturessets of images 123 that include the gray level images as indicated in ablock 602. In one embodiment, calibration unit 124 causes each projector112 to project a series of M gray levels from black to white where M isgreater than or equal to two, and camera 122 captures two images, images123C(N)(M) (shown in FIGS. 8 and 11) and 123D(N)(M) (shown in FIG. 10),of each gray level M for each projector 112(N). Camera 122 captures eachimage 123C with a relatively short exposure to detect the brightestlevels without saturation and each image 123D with a relatively longexposure to obtain usable image signal at the darkest levels. In someembodiments, camera 122 captures long-exposure images only forrelatively dark projector gray levels, so that the number of capturedimages 123C does not equal the number of captured images 123D. In otherembodiments, image sets 123C and 123D are combined into single set ofimagery 123 using high-dynamic range (HDR) imaging techniques so thatthe resulting set of images are not saturated and all have the samebrightness scale. In still other embodiments, only a single set ofimagery 123C is captured using either an intermediate exposure time or acamera capable of capturing non-saturated data over a large range ofscene brightnesses. Camera 122 captures all images 123C and 123D inthree-channel color. While gray levels for a first projector 112 arebeing captured, calibration unit 124 causes all other projectors 112that overlap the first projector on display surface 116 to be turned onand to project black.

Camera 122 may be operated in a linear output mode in capturing sets ofimages 123C and 123D to cause image values to be roughly proportional tothe light intensity at the imaging chip of camera 122. If camera 122does not have a linear output mode, the camera brightness response curvemay be measured by any suitable method and inverted to produce linearcamera image data.

In other embodiments, calibration unit 124 may cause any anothersuitable series of images to be projected and captured by camera 122.

Calibration unit 124 determines sets of inverse TRFs 700R, 700G, and700B (shown in FIG. 7) for each pixel location of each color plane ofeach projector 112 using a respective set of images 123C as indicated ina block 604. In one embodiment, the set of inverse TRFs 700R includesone inverse TRF for each pixel location in the red color plane of aprojector 112, the set of inverse TRFs 700G includes one inverse TRF foreach pixel location in the green color plane of a projector 112, and theset of inverse TRFs 700B includes one inverse TRF for each pixellocation in the blue color plane of a projector 112. In otherembodiments, each set of inverse TRFs 700R, 700G, and 700B includes oneinverse TRF for each set of pixel locations in a projector 112 whereeach set of pixel locations includes all pixel locations in a projector112 or a subset of pixel locations (e.g., pixel locations from selectedregions of projector 112) in a projector 112.

To determine the sets of inverse TRFs 700R, 700G, and 700B, calibrationunit 124 determines TRFs for each pixel location of each color plane ofeach projector 112 using the respective set of images 123C and geometricmeshes 404 and 414(i), where i is between 1 and N. In other embodiments,calibration unit 124 may determine sets of inverse TRFs 700R, 700G, and700B using other forms of geometric correction data that map cameralocations to projector frame buffer locations. Interpolation between themeasured gray levels in images 123C may be applied to obtain TRFs withproper sampling along the brightness dimension. Calibration unit 124then derives the sets of inverse TRFs 700R, 700G, and 700B from the setsof TRFs as described in additional detail below with reference to FIG.6C.

The generation of inverse TRFs is described herein for red, green, andblue color planes. In other embodiments, the inverse TRFs may begenerated for other sets of color planes.

Calibration unit 124 determines a blend map 702 (shown in FIG. 7) foreach projector 112 using a respective set of geometric meshes 304, 314,404, and 414(i) (i.e., the meshes between the screen domain, cameradomain, and the domain of projector 112(i), where i is between 1 and N,as described above) as indicated in a block 606. In other embodiments,calibration unit 124 may determine a blend map 702 using other forms ofgeometric correction data that map screen locations to projector framebuffer locations. Calibration unit 124 determines attenuating factors ineach blend map 702 that correspond to pixel locations in a respectiveimage frame 110 that fall within an overlap region in an image 114 ondisplay surface 116 with at least one other image 114 from at least oneother frame 110. Accordingly, each attenuating factor is configured toattenuate a corresponding pixel value in a pixel location of image frame110 in the process of generating a frame 110. The process of determiningblend maps 702 is described in additional detail below with reference toFIGS. 6D, 6E, and 9.

Calibration unit 124 determines an offset map 704 for each projector 112using a respective set of images 123D and respective geometric meshes304, 314, 404, and 414(i) as indicated in a block 608. In otherembodiments, calibration unit 124 may determine an offset map 704 usingother forms of geometric correction data that map screen locations toprojector frame buffer locations. Each offset map 704 includes a set ofoffset factors that are configured to be applied to a frame 110A togenerate smooth black levels across the display of an image 114. Theprocess of determining offset maps 704 is described in additional detailbelow with reference to FIGS. 6F and 10.

Calibration unit 124 determines a scale map 706 for each projector 112using a respective set of images 123C, respective blend maps 702, andrespective geometric meshes 304, 314, 404, and 414(i) as indicated in ablock 610. In other embodiments, calibration unit 124 may determine ascale map 706 using other forms of geometric correction data that mapscreen locations to projector frame buffer locations. Each scale map 706includes a set of attenuating factors that are configured to be appliedto a frame 110A to generate smooth brightness levels across the displayof an image 114. By forming each scale map 706 using a respective blendmap 702, scale maps 706 may be configured to increase the overallsmoothness of the brightness levels across the display of all images114. The process of determining scale maps 706 is described inadditional detail below with reference to FIGS. 6G and 11.

Photometric correction information 128 includes a blend map 702, anoffset map 704, and a scale map 706 for each projector 112 in oneembodiment. In other embodiments, photometric correction information 128may omit one or more of a blend map 702, an offset map 704, and a scalemap 706.

FIG. 6B illustrates a method of rendering a frame 110A using photometriccorrection information 128 to perform photometric correction on frame100A to generate a frame 110B. Frame generator 108 performs the methodof FIG. 6B for each frame 110A(1) through 110A(N), respectively, forprojection by projectors 112(1) through 112(N), respectively. Framegenerator 108 performs geometric correction on frames 110A, as describedabove in Section II, prior to performing the photometric correction ofFIGS. 6B and 7 in one embodiment. The method of FIG. 6B will bedescribed with reference to FIG. 7. FIG. 7 is a schematic diagramillustrating a process of rendering image frames 110A using photometriccorrection information 128.

Referring to FIGS. 6B and 7, frame generator 108 applies a gammafunction 712 to a frame 110A as indicated in a block 612. The gammafunction may be any suitable function (e.g., an exponential function)configured to prevent images 114 from appearing on display surface 116as saturated. Many display devices employ an exponential gamma functionin order to create imagery that is more perceptually pleasing and bettersuited to the logarithmic brightness response properties of the humaneye. The gamma function may be the same for each projector 112 or differbetween projectors 112.

Frame generator 108 applies a scale map 706 and a blend map 702 to aframe 110A as indicated in a block 614. More particularly, framegenerator 108 multiplies the pixel values of frame 110A withcorresponding scale factors in scale map 706 and blend map 702 asindicated by a multiplicative function 714. In one embodiment, framegenerator 108 combines scale map 706 and blend map 702 into a singleattenuation map 708 (i.e., by multiplying the scale factors of scale map706 by the attenuation factors of blend map 702) and applies attenuationmap 708 to frame 110A by multiplying the pixel values of frame 110A withcorresponding attenuation factors in attenuation map 708. In otherembodiments, frame generator 108 applies scale map 706 and blend map 702separately to frame 110A by multiplying the pixel values of frame 110Awith one of corresponding scale factors in scale map 706 orcorresponding attenuation factors in blend map 702 and then multiplyingthe products by the other of the corresponding scale factors in scalemap 706 or corresponding attenuation factors in blend map 702. Bymultiplying pixel values in frame 110A by attenuating factors from scalemap 706 and blend map 702, frame generator 108 reduces the brightness ofselected pixel values to smooth the brightness levels of a correspondingimage 114.

Frame generator 108 applies an offset map 704 to a frame 110 asindicated in a block 616. Frame generator 108 adds the offset factors ofoffset map 704 to corresponding pixel values in frame 110 as indicatedby an additive function 716. By adding pixel values in frame 110 withoffset factors from offset map 704, frame generator 108 increases thebrightness of selected pixel values to smooth the black level of thecombination of projected images 114 across display surface 116.

Frame generator 108 applies sets of inverse TRFs 700R, 700G, and 700B toa frame 110A to generate a frame 110B as indicated in a block 618. Framegenerator 108 applies inverse TRF 700R to the red color plane of a frame110A, the inverse TRF 700G to the green color plane of a frame 110A, andthe inverse TRF 700B to the blue color plane of a frame 110A to convertthe pixel values in a frame 110. Frame generator 108 provides frame 110to a corresponding projector 112.

In one embodiment, the above corrections may be combined into a single3D lookup table (e.g., look-up tables 806R, 806G, and 806B shown in FIG.8) with two spatial dimensions and one brightness dimension for eachcolor plane. Each 3D lookup table incorporates black offset, brightnessattenuation, and application of the set of inverse TRFs for that colorplane.

Projector 112 projects frame 110B onto display surface 116 to form image114 as indicated in a block 210. The remaining projectors 112simultaneously project corresponding frames 110B to form the remainingimages 114 on display surface 116 with geometric and photometriccorrection. Accordingly, the display of images 114 appears spatiallyaligned and seamless with smooth brightness levels across thecombination of projected images 114 on display surface 116.

FIG. 6C illustrates a method for performing the function of block 604 ofFIG. 6A. Namely, the method of FIG. 6C illustrates one embodiment ofdetermining the sets of inverse TRFs 700R, 700G, and 700B for aprojector 112. Calibration unit 124 performs the method of FIG. 6C foreach set of captured image frames 123C(1) through 123C(N) to generatecorresponding sets of inverse TRFs 700R, 700G, and 700B for projectors112(1) through 112(N), respectively. The method of FIG. 6C will bedescribed with reference to FIG. 8. FIG. 8 is a block diagramillustrating a process of determining inverse tone reproductionfunctions for each color plane of a projector 112.

The generation of the sets of inverse TRFs 700R, 700G, and 700B will bedescribed for red, green, and blue color planes. In other embodiments,the sets of inverse TRFs may be generated for other sets of colorplanes.

Referring to FIGS. 6C and 8, calibration unit 124 converts a set ofcaptured camera images 123C into a projector coordinate domain of aprojector 112 as indicated in a block 622. As shown in FIG. 8,calibration unit 124 geometrically warps the set of captured images123C(1) to 123C(M) into converted images 800(1) to 800(M) using mesh 404in the camera domain and the respective mesh 414(i) in the domain ofprojector 112 in one embodiment. In other embodiments, calibration unit124 maps the set of captured images 123C(1) to 123C(M) into thecoordinate domain of projector 112 in any other suitable way.

Calibration unit 124 generates a set of curves for each color plane of aprojector 112 by plotting, for a selected set of pixel locations of aprojector 112, gray level values projected by a projector 112 versusprojector output brightness values measured by a camera at correspondingpixel locations in the set of converted images 800 as indicated in ablock 624. The selected set of pixel locations may include all of thepixel locations in projector 112, a subset of pixel locations inprojector 112, or a single pixel location in projector 112.

As shown in FIG. 8, calibration unit 124 generates sets of TRFs 804R,804G, and 804B for each pixel value in the red, green, and blue colorplanes, respectively, from gray level input values 802(1) through 802(M)projected by a respective projector 112 and from the corresponding setof brightness measurements contained in converted images 800(1) through800(M) for the selected set of pixel locations of projector 112. Toaccount for spatial variations in projector 112, the selected set ofpixel locations of projector 112 may include all of the pixel locationsof projector 112 or a set of pixel locations of projector 112distributed throughout the domain of projector 112.

Calibration unit 124 normalizes the domain and range of each curve ineach set of curves to [0, 1] as indicated in a block 626, and invertsthe domain and range of each curve in each set of curves as indicated ina block 628. The inverted curves form inverse TRFs 700R, 700G, and 700B.In one embodiment, calibration unit 124 generates a separate inverse TRFfor each pixel location for each color plane in the domain of projector112. In other embodiments, calibration unit 124 may average a set of thenormalized and inverted curves to form one inverse TRF 700R, 700G, and700B for all or a selected set of pixel locations in each color plane.

Calibration unit 124 converts the inverted curves into any suitablerender format as indicated in a block 630. In one embodiment,calibration unit 124 determines sets of functional fit parameters 808R,808G, and 808B that best fit each inverse TRF 700R, 700G, and 700B to afunctional form such as an exponential function. The fit parameters808R, 808G, and 808B are later applied together with the functional formby frame generator 108 to render frames 110B to compensate for thenon-linearity of the transfer functions of projectors 112.

In other embodiments, calibration unit 124 generates look-up tables806R, 806G, and 806B from the sets of inverse tone reproductionfunctions 700R, 700G, and 700B. In one form, calibration unit 124generates each look-up table 806R, 806G, and 806B as a three dimensionaltable with a different set of values for corresponding color values ateach coordinate location of projector 112 for each color plane accordingto sets of inverse tone reproduction functions 700R, 700G, and 700B. Inother forms, calibration unit 124 generates each look-up table 806R,806G, and 806B as a one dimensional table with the same set or subset ofvalues for corresponding color values at each coordinate location ofprojector 112 according to sets of inverse tone reproduction functions700R, 700G, and 700B. The lookup tables are later applied by framegenerator 108 to render frames 110B to compensate for the non-linearityof the transfer functions of projectors 112.

FIG. 6D illustrates a method for performing a portion of the function ofblock 606 of FIG. 6A. Namely, the method of FIG. 6D illustrates oneembodiment of determining blend maps for use in generating attenuationmaps. The method of FIG. 6D will be described with reference to FIGS. 9Aand 9B. FIGS. 9A and 9B are schematic diagrams illustrating a process ofdetermining blend maps.

Referring to FIGS. 6D, 9A and 9B, calibration unit 124 identifiesoverlapping regions of projectors 112 using geometric meshes 304, 314,404, and 414(i) as indicated in a block 642. To do so, calibration unit124 identifies pixel locations in each projector 112 that correspond tothe same screen locations in the screen domain as other pixel locationson one or more other projectors 112 using geometric meshes 304, 314,404, and 414(i). The set of screen locations forms the overlap regionsin the screen domain, and the corresponding pixel locations for eachprojector 112 form the overlap regions in the projector domains.

In an example shown in FIG. 9A, frames 110A(1) through 110A(6) arerepresented in the screen domain subsequent to being geometricallycorrected as described above with reference to Section II. Frames110A(1) and 110A(2) form an overlap region 900, frames 110A(2), 110A(3),and 110A(4) form an overlap region 902, and frames 110A(3), 110A(4),110A(5), and 110A(6) form an overlap region 906. These overlap regions900, 902, and 904 in the screen domain correspond to overlap regions150, 152, and 154 (shown in FIG. 1D) on display surface 116. Otheroverlap regions in the screen domain are shown in other shaded regionsof FIG. 9A. Referring to FIG. 9B, calibration unit 124 identifiesregions 910A and 910B in the projector coordinate domains of projectors112(1) and 112(2), respectively, that correspond to overlap region 900in the screen domain.

Calibration unit 124 generates a blend map 702 for each projector 112with an attenuation factor for each pixel location located within theoverlapping regions as indicated in a block 644. Referring to FIG. 9B,for each pixel location in region 910A of projector coordinate domainP(1), calibration unit 124 determines an attenuation factor in blend map702(1). For example, for pixel location 912 in region 910A, calibrationunit 124 determines an attenuation factor for a corresponding location922(1) in blend map 702(1) as indicated by a dashed arrow 916(1). Theattenuation factor in location 922(1) corresponds to the screen location900A (FIG. 9A). Similarly, for each pixel location in region 910B ofprojector coordinate domain P(2), calibration unit 124 determines anattenuation factor in blend map 702(2). Thus, calibration unit 124determines an attenuation factor for a location 922(2) in blend map702(2) that corresponds to pixel location 914 in region 910B asindicated by a dashed arrow 916(2). The attenuation factor in location922(2) also corresponds to the screen location 900A (FIG. 9A).

In one embodiment, calibration unit 124 generates each attenuationfactor to be in the range of zero to one. In this embodiment,calibration unit 124 generates the attenuation factors that correspondto a screen location across all blend maps 702 such that the sum of theattenuation factors corresponding to any screen location is equal toone. Thus, in the example of FIG. 9B, the sum of the attenuation factorof location 922(1) and the attenuation factor of location 922(2) isequal to one. In other embodiments, calibration unit 124 may generateeach attenuation factor to be in any other suitable range of values.

FIG. 6E illustrates one embodiment of determining attenuation factorsfor blend maps 702 for a screen location as referenced in block 644 ofFIG. 6D. Calibration unit 124 performs the method of FIG. 6E for screenlocations in overlapping regions in the screen domain in one embodiment.

In FIG. 6E, calibration unit 124 determines at least two distancesbetween a first pixel location in a first frame 110A and edges of thefirst frame 110A as indicated in a block 648. In FIG. 9B, for example,calibration unit 124 determines a distance d(1)A between pixel location912 and edge 110A(1)A, distance d(1)B between pixel location 912 andedge 110A(1)B, a distance d(1)C between pixel location 912 and edge110A(1)C, and a distance d(1)D between pixel location 912 and edge110A(1)D.

Calibration unit 124 determines at least two distances between a secondpixel location in a second frame 110A and edges of the second frame 110Aas indicated in a block 650. In FIG. 9B, for example, calibration unit124 determines a distance d(2)A between pixel location 914 and edge110A(2)A, distance d(2)B between pixel location 914 and edge 110A(2)B, adistance d(2)C between pixel location 914 and edge 110A(2)C, and adistance d(2)D between pixel location 914 and edge 110A(2)D.

Calibration unit 124 determines whether there is another overlappingframe 110A as indicated in a block 652. If there is not anotheroverlapping frame 110A, as in the example of FIG. 9B, then calibrationunit 124 determines attenuation factors for blend maps 702 correspondingto the pixel locations in the first and second frames 110A as indicatedin a block 656. Calibration unit 124 determines each attenuation factoras a proportion of the sum of the respective products of the distancesbetween pixel locations in respective frames 110A and the edges of therespective frames 110A using Equations 3 and 4.

$\begin{matrix}{{G\left( {\overset{\rightarrow}{p}}_{i} \right)} = \frac{ɛ_{i}}{\sum\limits_{j = 1}^{N}ɛ_{j}}} & (3) \\{ɛ_{i} = {\prod\limits_{k = 1}^{x}d_{i,k}}} & (4)\end{matrix}$

In Equations 3 and 4, i refers to the ith projector 112 and k refers tothe number of calculated distances for each pixel location in arespective frame 110A where k is greater than or equal to 2. Equation 3,therefore, is used to calculate each attenuation factor as a ratio of aproduct of distances calculated in a given frame 110A to a sum of theproduct of distances calculated in the given frame 110A and the productor products of distances calculated in the other frame or frames 110Athat overlap with the given frame 110A.

In addition, ε_(i)({right arrow over (p)}_(i)) forms a scalar-valuedfunction over projector coordinates where ε_(i)({right arrow over(p)}_(i)) goes to zero as {right arrow over (p)}_(i) approaches any edgeof a projector 112, and ε_(i)({right arrow over (p)}_(i)) and thespatial derivative of ε_(i)({right arrow over (p)}_(i)) are notdiscontinuous anywhere inside the coordinate bounds of the projector112.

Using Equations 3 and 4, calibration unit 124 calculates the attenuationfactor for location 922(1) in FIG. 9B by dividing the product ofdistances d(1)A, d(1)B, d(1)C, and d(1)D with the sum of the product ofdistances d(1)A, d(1)B, d(1)C, and d(1)D and the product of distancesd(2)A, d(2)B, d(2)C, and d(2)D. Similarly, calibration unit 124calculates the attenuation factor for location 922(2) in FIG. 9B bydividing the product of distances d(2)A, d(2)B, d(2)C, and d(2)D withthe sum of the product of distances d(1)A, d(1)B, d(1)C, and d(1)D andthe product of distances d(2)A, d(2)B, d(2)C, and d(2)D.

Calibration unit 124 stores the attenuation factors in respective blendmaps 702 as indicated in a block 658. In FIG. 9B, calibration unit 124stores the attenuation factor for pixel location 912 in frame 110A(1) inlocation 922(1) of blend map 702(1) and the attenuation factor for pixellocation 914 in frame 110A(2) in location 922(2) of blend map 702(2).

In the example of FIG. 9B, calibration unit 124 repeats the method ofFIG. 6E for each pixel location in overlapping regions 910A and 910B todetermining the remaining attenuation factors in regions 924(1) and924(2) of blend maps 702(1) and 702(2) respectively.

For pixel locations in regions of frames 110A that, when appearing aspart of projected image 114 on display surface 116, do not overlap withany projected images 114 projected by other projectors 112, calibrationunit 124 sets the attenuation factors in corresponding regions of blendmaps 702 to one or any other suitable value to cause images 114 not tobe attenuated in the non-overlapping regions on display surface 116. Forexample, calibration unit 124 sets the attenuation factors of all pixelsin regions 926(1) and 926(2) of blend maps 702(1) and 702(2),respectively, to one so that blend maps 702(1) and 702(2) do notattenuate corresponding pixel locations in frames 110A(1) and 110A(2)and corresponding screen locations on display surface 116.

Referring back to block 652 of FIG. 6E, if calibration unit 124determines that there is one or more additional overlapping frames 110A,then calibration unit 124 determines at least two distances between eachadditional overlapping pixel location in each additional overlappingframe 110A and respective edges of each overlapping frame 110A asindicated in a block 654.

In region 902 of FIG. 9A, for example, calibration unit 124 determinesat least two distances for each corresponding pixel location in frames110A(2), 110A(3) and 110A(4) and uses the three sets of distances inEquations 3 and 4 to determine attenuation factors corresponding to eachpixel location for blend maps 702(2), 702(3) (not shown), and 702(4)(not shown).

Likewise in region 904 of FIG. 9A, for example, calibration unit 124determines at least two distances for each corresponding pixel locationin frames 110A(3), 110A(4), 110A(5) and 110A(6) and uses the four setsof distances in Equations 3 and 4 to determine attenuation factorscorresponding to each pixel location for blend maps 702(3) (not shown),702(4) (not shown), 702(5) (not shown), and 702(6) (not shown).

In embodiments where k is equal to four as in the example of FIG. 9B(i.e., four distances are calculated for each pixel location in a frame110A), calibration unit 124 calculates all four distances between pixellocations in overlapping frames 110A and the respective edges of frames110A and uses all four distances from each overlapping frame 110A inEquations 3 and 4 to calculate each attenuation factor.

In other embodiments, k is equal to two (i.e., two distances arecalculated for each pixel location in a frame 110A). In embodimentswhere k is equal to two, calibration unit 124 uses the two shortestdistances between pixel locations in overlapping frames 110A and therespective edges of frames 110A in Equations 3 and 4. To determine theshortest distances, calibration unit 124 may calculate all fourdistances between a pixel location in a frame 110A and the respectiveedges of frame 110A for each of the overlapping frames 110A and selectthe two shortest distances for each frame 110A for use in Equations 3and 4.

FIG. 6F illustrates a method for performing a portion of the function ofblock 606 of FIG. 6A. Namely, the method of FIG. 6F illustrates oneembodiment of determining offset maps. The method of FIG. 6F will bedescribed with reference to FIG. 10. FIG. 10 is a block diagramillustrating a process of determining offset maps.

Referring to FIGS. 6F and 10, calibration unit 124 generates a blacklevel measurement map 1002 from the set of captured images 123D andgeometric meshes 304, 314, 404, and 414(i) as indicated in a block 662.The spatial dimensions of black level measurement map 1002 may beselected independently of the characteristics of captured images 123Dand geometric meshes 304, 314, 404, and 414(i), so that black levelmeasurement map 1002 may contain an arbitrary number of pixels.Calibration unit 124 maps black measurement values from the set ofcaptured images 123D into the screen coordinate domain using geometricmeshes 304, 314, 404, and 414(i) to generate black level measurement map1002. Accordingly, black level measurement map 1002 may include a blacklevel measurement value determined from the set of captured images 123Dfor each pixel that corresponds to a screen location on display surface116.

Calibration unit 124 applies a smoothing function 1004 to black levelmeasurement map 1002 to generate a black level target map 1006 asindicated in a block 664. Calibration unit 124 derives black leveltarget map 1006 from black level measurement map 1002 such that blacklevel target map 1006 is spatially smooth across the display of images114 on display surface 116.

In one embodiment, smoothing function 1004 represents an analogousversion of the constrained gradient-based smoothing method applied tosmooth brightness levels in “Perceptual Photometric Seamlessness inProjection-Based Tiled Displays”, A. Majumder and R. Stevens, ACMTransactions on Graphics, Vol. 24., No. 1, pp. 118-139, 2005 which isincorporated by reference herein. Accordingly, calibration unit 124analogously applies the constrained gradient-based smoothing methoddescribed by Majumder and Stevens to the measured black levels in blacklevel measurement map 1002 to generate black level target map 1006 inthis embodiment.

In one embodiment of the constrained gradient-based smoothing method,pixels in black level target map 1006 corresponding to locations ondisplay surface 116 covered by projected images 114 are initialized withcorresponding pixel values from black level measurement map 1002. Allpixels in black level target map 1006 corresponding to locations ondisplay surface 116 not covered by projected images 114 are initializedto a value lower than the minimum of any of the pixels of black levelmeasurement map 1002 corresponding to areas of display surface 116covered by projected images 114. The pixels of black level target map1006 are then visited individually in four passes through the image thatfollow four different sequential orderings. These four orderings are 1)moving down one column at a time starting at the left column and endingat the right column, 2) moving down one column at a time starting at theright column and ending at the left column, 3) moving up one column at atime starting at the left column and ending at the right column, and 4)moving up one column at a time starting at the right column and endingat the left column. During each of the four passes through the image, ateach pixel the value of the pixel is replaced by the maximum of thecurrent value of the pixel and the three products formed by multiplyingeach of the three adjacent pixels already visited on this pass byweighting factors. The weighting factors are less than one and enforcespatial smoothness in the resulting black level target map 1006, withhigher weighting factors creating a more smooth result. The weightingfactors may be derived in part from consideration of the human contrastsensitivity function, the expected distance of the user from the displaysurface 116, and the resolution of the projected images 114. Thisprocess is repeated independently for each color plane of black leveltarget map 1006.

Calibration unit 124 generates an offset map 704 for each projector 112using black level measurement map 1002, black level target map 1006, andthe camera images 123D captured with relatively long exposure time asindicated in a block 666. Calibration unit 124 generates a set of offsetvalues in each offset map 704 by first subtracting values in blackoffset measurement map 1002 from corresponding values in black leveltarget map 1006 to generate sets of difference values. Calibration unit124 divides each difference value in each set of difference values bythe numbers of projectors 112 that project onto the screen locationsthat correspond to the respective difference values to generate sets ofdivided values. Calibration unit 124 interpolates between measuredbrightnesses at corresponding locations in captured images 123D todetermine the projector inputs required to produce the divided values,and these projector inputs are used as the sets of offset values inoffset maps 704. That is, at each pixel location in offset map 704, thecorresponding location in images 123D is determined, and the measuredbrightnesses in 123D for different gray level inputs to correspondingprojector 112 are examined to find the two images 123D whose measuredbrightnesses at this location bound above and below the correspondingdivided value. Interpolation is performed on the projector input graylevels corresponding to these two images 123D to estimate the projectorinput required to produce the divided value. The estimated projectorinput is stored at the corresponding location in black offset map 704.In other embodiments, calibration unit 124 performs interpolation inother ways such as by using more than two images 123D.

FIG. 6G illustrates a method for performing a portion of the function ofblock 606 of FIG. 6A. Namely, the method of FIG. 6G illustrates oneembodiment of determining attenuation maps. The method of FIG. 6G willbe described with reference to FIG. 11. FIG. 11 is a block diagramillustrating a process of determining attenuation maps.

Referring to FIGS. 6G and 11, calibration unit 124 generates a whitelevel measurement map 1102 from the set of captured images 123C,geometric meshes304, 314, 404, and 414(i), blend maps 702, and blacklevel measurement map 1002 as indicated in a block 672. White levelmeasurement map 1102 contains white level measurement values that eachidentify the maximum brightness level at a corresponding location ondisplay surface 116 after blend maps 702 have been applied, asdetermined from the set of captured images 123C and blend maps 702.Accordingly, white level measurement map 1102 identifies brightnesslevels at screen locations across the display of images 114.

Calibration unit 124 maps measurement values in the set of capturedimages 123C into the screen coordinate domain using geometric meshes304, 314, 404, and 414(i) to generate the white level measurement valuesin white level measurement map 1102. Calibration unit 124 then subtractsblack level measurement values in black level measurement map 1002 fromcorresponding white level measurement values in white level measurementmap 1102 to remove the black offset from white level measurement map1102. Calibration unit 124 next applies blend maps 702 to white levelmeasurement map 1102 by multiplying white level measurement values bycorresponding attenuation factors of blend maps 702 to attenuate pixelvalues in the overlap regions of white level measurement map 1102.Accordingly, white level measurement map 1102 includes a set of whitelevel measurement values from the set of captured images 123C for eachscreen location on display surface 116 that are adjusted bycorresponding black level offset measurements in black level measurementmap 1002 and corresponding attenuation factors in blend maps 702.

Calibration unit 124 applies a smoothing function 1104 to white levelmeasurement map 1102 to generate a white level target map 1106 asindicated in a block 674. White level target map 1106 represents adesired, smooth white (maximum brightness) level across the display ofimages 114 on display surface 116.

In one embodiment, smoothing function 1104 represents the constrainedgradient-based smoothing method applied to smooth brightness levels in“Perceptual Photometric Seamlessness in Projection-Based TiledDisplays”, A. Majumder and R. Stevens, ACM Transactions on Graphics,Vol. 24., No. 1, pp. 118-139, 2005 which is incorporated by referenceherein. Accordingly, calibration unit 124 applies the constrainedgradient-based smoothing method described by Majumder and Stevens to themeasured white levels in white level measurement map 1102 to generatewhite level target map 1106.

In one embodiment of the constrained gradient-based smoothing method,pixels in white level target map 1106 corresponding to locations ondisplay surface 116 covered by projected images 114 are initialized withcorresponding pixel values from white level measurement map 1102. Allpixels in white level target map 1106 corresponding to locations ondisplay surface 116 not covered by projected images 114 are initializedto a value higher than the minimum of any of the pixels of black levelmeasurement map 1102 corresponding to areas of display surface 116covered by projected images 114. The pixels of white level target map1106 are then visited individually in four passes through the image thatfollow four different sequential orderings. These four orderings are 1)moving down one column at a time starting at the left column and endingat the right column, 2) moving down one column at a time starting at theright column and ending at the left column, 3) moving up one column at atime starting at the left column and ending at the right column, and 4)moving up one column at a time starting at the right column and endingat the left column. During each of the four passes through the image, ateach pixel the value of the pixel is replaced by the minimum of thecurrent value of the pixel and the three products formed by multiplyingeach of the three adjacent pixels already visited on this pass byweighting factors. The weighting factors are greater than one andenforce spatial smoothness in the resulting white level target map 1106,with lower weighting factors creating a more smooth result. Theweighting factors may be derived in part from consideration of the humancontrast sensitivity function, the expected distance of the user fromthe display surface 116, and the resolution of the projected images 114.This process is repeated independently for each color plane of whitelevel target map 1106.

Calibration unit 124 generates a scale map 706 for each projector 112using white level measurement map 1102, white level target map 1106, andblack level target map 1006 as indicated in a block 676. Calibrationunit 124 generates a set of scale factors in each scale map 706 by firstsubtracting values in white attenuation target map 1006 fromcorresponding values in black level target map 1006 to generate sets ofdifference values. Calibration unit 124 divides each difference value ineach set of difference values by corresponding values in white levelmeasurement map 1102 to generate sets of scale factors in scale maps706.

Calibration unit 124 generates an attenuation map 708 for each projector112 using a respective scale map 706 and a respective blend map 702 asindicated in a block 678. Calibration unit 124 generates a set ofattenuation factors in each attenuation map 708 by multiplying acorresponding set of scale factors from a corresponding scale map 706 bya corresponding set of attenuation factors from a corresponding blendmap 702.

The derivation of offset maps 702 and attenuation maps 708 will now bedescribed. Let I({right arrow over (s)}) be the three-channel color ofan input image 102 to be displayed at screen location {right arrow over(s)}. By Equation 1, this is also the color corresponding to projectorcoordinate {right arrow over (p)}_(i)=P_(i)({right arrow over (s)}) inimage frame 110A. If it is assumed that the ith projector 112's TRF hasbeen linearized by application of inverse TRF h⁻¹(I_(i,l)) (e.g., byapplication of the sets of inverse TRFs 700R, 700G, and 700B), where lindicates the color plane in a set of color planes (e.g., RGB), then theprojector output color L({right arrow over (p)}_(i)) at pixel location{right arrow over (p)}_(i) is as shown in Equation 5.

L({right arrow over (p)} _(i))=[G({right arrow over (p)} _(i))(W({rightarrow over (p)} _(i))−B({right arrow over (p)} _(i)))]*I(P _(i)({rightarrow over (s)}))+B({right arrow over (p)} _(i))   (5)

This is the equation of a line that, over the domain of I=[0, 1], has aminimum value at I=0 equal to the measured black offset B({right arrowover (p)}_(i)) at the screen location corresponding to {right arrow over(p)}_(i), and a maximum value at I=1 equal to the measured white offsetat the screen location corresponding to {right arrow over (p)}_(i) afterattenuation by geometric blend function G({right arrow over (p)}_(i))(e.g., by using the attenuation factors in blend maps 702).

To compensate for the linearity of the projector response, the inputimage color I is enhanced with an exponential function H (i.e., gammafunction 712 in FIG. 7), with an exponent typically around 2.3. Becauseof projector 112's linearity, H becomes the effective “gamma” of theentire image display system 100 and is controllable in software in oneembodiment. This enhancement is applied prior to other photometriccorrections and is expressed through a substitution in the aboveEquation 5 as shown in Equation 6.

L({right arrow over (p)} _(i))=[G({right arrow over (p)} _(i))(W({rightarrow over (p)} _(i))−B({right arrow over (p)} _(i)))]*H(I)+B({rightarrow over (p)} _(i))   (6)

For N projectors 112 overlapping at screen location {right arrow over(s)} on display surface 116, the expected output color on displaysurface 116 is obtained by summing Equation 6 across all projectors 112as shown in Equation 7.

$\begin{matrix}{{L\left( \overset{\rightarrow}{s} \right)} = {{{H(I)}*{\sum\limits_{i = 1}^{N}\left\lbrack {{G\left( {\overset{\rightarrow}{p}}_{i} \right)}\left( {{W\left( {\overset{\rightarrow}{p}}_{i} \right)} - {B\left( \overset{\rightarrow}{s} \right)}} \right)} \right\rbrack}} + {B\left( \overset{\rightarrow}{s} \right)}}} & (7)\end{matrix}$

For I=0 and I=1, L({right arrow over (s)}) equates to black and whitemeasurement map values B({right arrow over (s)}) and W({right arrow over(s)}), respectively.

The desired projector response at {right arrow over (s)}, defined byblack level and white level target maps 1006 and 1106, respectively,computed as described above, is also a line, but with a different slopeand intercept as shown in Equation 8.

L({right arrow over (s)})=H(I)*(W _(t)({right arrow over (s)})−B_(t)({right arrow over (s)}))+B _(t)({right arrow over (s)})   (8)

Equations 7 and 8 are brought into agreement by inserting into Equation7 a scale factor α({right arrow over (p)}_(i)) and offset factorβ({right arrow over (p)}_(i)) that are the same at all coordinates{right arrow over (p)}_(i) corresponding to screen location {right arrowover (s)} for all projectors 112 overlapping at screen location {rightarrow over (s)} as shown in Equation 9.

$\begin{matrix}{{L\left( \overset{\rightarrow}{s} \right)} = {{{H(I)}*{\sum\limits_{i = 1}^{N}\left\lbrack {{\alpha \left( {\overset{\rightarrow}{p}}_{i} \right)}{G\left( {\overset{\rightarrow}{p}}_{i} \right)}\left( {{W\left( {\overset{\rightarrow}{p}}_{i} \right)} - {B\left( \overset{\rightarrow}{s} \right)}} \right)} \right\rbrack}} + \left( {{\beta \left( {\overset{\rightarrow}{p}}_{i} \right)} + {B\left( \overset{\rightarrow}{s} \right)}} \right)}} & (9)\end{matrix}$

Equations 10 and 11 cause Equations 8 and 9 to be equal.

$\begin{matrix}{{\alpha \left( {\overset{\rightarrow}{p}}_{i} \right)} = \frac{{W_{t}\left( \overset{\rightarrow}{s} \right)} - {B_{t}\left( \overset{\rightarrow}{s} \right)}}{\sum\limits_{i = 1}^{N}{{G\left( {\overset{\rightarrow}{p}}_{i} \right)}\left( {{W\left( {\overset{\rightarrow}{p}}_{i} \right)} - {B\left( \overset{\rightarrow}{s} \right)}} \right)}}} & (10) \\{{\beta \left( {\overset{\rightarrow}{p}}_{i} \right)} = \frac{{B_{t}\left( \overset{\rightarrow}{s} \right)} - {B\left( \overset{\rightarrow}{s} \right)}}{N}} & (11)\end{matrix}$

Intuitively, the value of α({right arrow over (p)}_(i)) at a givenscreen location is the ratio of the target display dynamic range here(from the smoothed white level target map 1106 (W_(t)) down to thesmoothed black level target map 1006 (B_(t))) to the original measureddynamic range of the tiled display after geometric blending has beenapplied. β({right arrow over (p)}_(i)) distributes the differencebetween black level target map 1006 B_(t) and black level measurementmap 1002 B equally among projectors 112 overlapping at {right arrow over(s)}. Offset maps 704 used by frame generator 108 are described byβ({right arrow over (p)}_(i)), while attenuation maps 708 are describedby α({right arrow over (p)}_(i))*G({right arrow over (p)}_(i)). BecauseB, B_(t), W, and W_(t), are all in three-channel color, the above methodcan produce separate results for each color channel.

Application of geometric blending using blend maps 702 during creationof white level measurement map 1102 W({right arrow over (s)}) and priorto the creation of white level target map 1106 W_(t)({right arrow over(s)}) may result in photometric calibration that is more tolerant ofgeometric calibration error. A white measurement map created withoutgeometric blending may contain sharp brightness discontinuities atprojector overlap region boundaries. In contrast, the method describedherein blends projector contributions in overlap regions to produce arelatively smooth white level measurement map 1102 W({right arrow over(s)}) whose differences from uniformity reflect only the intrinsicbrightness variations of projectors 112, rather than spatial overlapgeometry. Elimination of discontinuities in white level measurement map1102 (W({right arrow over (s)})) through geometric blending may yieldsmoother attenuation maps and allow for greater tolerance of geometriccalibration imprecision.

IV. Projection of Multiple Image Streams

In one form of the invention, image display system 100 (FIG. 1) isconfigured to simultaneously project multiple different image streams orvideo streams on display surface 116. In addition to simply displayingthe different streams in fixed locations on the surface 116, thelocation, display size, and other properties of the streams can betransformed dynamically and in real time in one embodiment. The dynamicrepositioning and rescaling of streams provided by one embodiment of theinvention allows one or more streams to be brought to emphasis at akeystroke by a user. The dynamic reconfiguration of projected streamsaccording to one form of the present invention is described in furtherdetail below with reference to FIGS. 12-15.

FIG. 12 is a block diagram illustrating the processing system 101 shownin FIG. 1A as configured for providing dynamically reconfigurablemultiple stream rendering according to one embodiment of the presentinvention. As shown in FIG. 12, processing system 101 includes memory1202, two central processing units (CPUs) 1210 and 1212, two graphicalprocessing units (GPUs) 1214 and 1216, user interface device 1218, andprocessing system display 1220. In one embodiment, processing system 101is a Hewlett-Packard xw9300 workstation, which includes two AMD Opteron2.19 GHz CPUs 1210 and 1212 and two Nvidia Quadro FX3400 GPUs 1214 and1216, each of which can drive two projectors 112 (FIG. 1A). In anotherembodiment, processing system 101 includes one or more additional GPUs,such as GPU 1217, which allows processing system 101 to drive more thanfour projectors 112. Additional projectors 112 can also be driven byusing multiple processing systems 101.

In one embodiment, user interface device 1218 is a mouse, a keyboard, orother device that allows a user to enter information into and interactwith processing system 101. In one embodiment, display 1220 is a cathoderay tube (CRT) display, flat-panel display, or any other type ofconventional display device. In another embodiment, processing system101 does not include a processing system display 1220. Memory 1202stores a plurality of different streams 1204(1)-1204(M) (collectivelyreferred to as streams 1204), multimedia framework 1206, and streamprocessing software modules 1208. In one embodiment, streams 1204 aredifferent video streams (e.g., the image content of each stream 1204 isdifferent than the content of the other streams 1204) with or withoutassociated audio streams. Geometric meshes 126 and photometriccorrection information 128 are stored in GPUs 1214 and 1216. In oneembodiment, processing system 101 processes streams 1204 based ongeometric meshes 126, photometric correction information 128, and userinput (e.g., stream selection, transformation or modificationparameters) entered via user interface device 1218, to generatecomposite or processed streams 1222(1)-1222(N) (collectively referred toas processed streams 1222), which are provided to projectors 112 forsimultaneous projection onto display surface 116. In another embodiment,rather than, or in addition to, relying on user input, processing system101 is configured to automatically generate stream modification ortransformation parameters. In one embodiment, the number M of streams1204 is equal to the number N of streams 1222. In other embodiments, thenumber M of streams 1204 is greater than or less than the number N ofstreams 1222. Processing system 101 is described in further detail belowwith reference to FIGS. 13-15.

FIGS. 13A-13C are diagrams illustrating a simplified representation ofthe simultaneous projection of multiple different streams 1302(1) to1302(6) (collectively referred to as displayed or projected streams1302) by display system 100 (FIG. 1A), and the dynamic reconfigurationof the projected streams 1302 according to one form of the presentinvention. In one embodiment, projected streams 1302 are video streams,and one or more of the projected streams 1302 may include an associatedaudio stream. Each projected stream 1302 corresponds to one of thestreams 1204 shown in FIG. 12. Streams 1204 are processed by processingsystem 101, including potentially combining multiple streams 1204 orportions of multiple streams 1204, to generate processed streams 1222,which are then projected by the projectors 112 onto display surface 116to generate the projected streams 1302. In one embodiment, displaysurface 116 is a non-planar developable display surface.

In one embodiment, the six different displayed or projected streams 1302are generated by projecting the four processed streams 1222 with fourprojectors 112 configured in a tiled arrangement to cover substantiallythe entire display surface 116. Six different streams 1204 are combinedby processing system 101 into the four processed streams 1222 forprojection by the four projectors 112. In another embodiment, more orless than four projectors 112 are used to produce the six differentstreams 1302. In one form of the invention, the display surface 116 istreated by processing system 101 as a single virtual display andmultiple-stream content can be shown on the display surface 116independent of the number of physical projectors 112 making up thedisplay.

The projected streams 1302 can originate from any arbitrary videosource. These sources can be local sources that are included in orcoupled directly to processing system 101, and can be remote sources.The streams can arrive at varying rates at the processing system 101,and do not need to be synchronized with other streams being displayed.Live streams can be shown by display system 100 with very low latency.

As shown in FIG. 13A, six video streams 1302 are simultaneouslyprojected onto display surface 116. The six projected video streams 1302shown in FIG. 13A are initially positioned in two rows and three columnswith no overlap between projected streams 1302, and the projected videostreams 1302 have the same size as each other (i.e., the projected videostreams 1302 each occupy substantially the same amount of area on thesurface 116). The locations and sizes of the projected video streams1302 shown in FIG. 13A represent “home” locations and sizes of thestreams 1302 according to one embodiment. The home locations and sizesare used in one embodiment when none of the projected video streams 1302is being individually emphasized by a user. By using user interfacedevice 1218 and display 1220, a user interacts with processing system101 to modify characteristics of one or more of the projected videostreams 1302, including moving or repositioning selected ones of thestreams 1302, and rescaling or changing the display size of selectedones of the streams 1302.

FIG. 13B shows the six projected video streams 1302 shown in FIG. 13Aafter a set of movement and resealing operations have been performed. Asshown in FIG. 13B, the projected video stream 1302(2) has been rescaledto be larger than the corresponding stream 1302(2) shown in FIG. 13A,and has been repositioned to the center of the display surface 116. Fiveof the projected video streams 1302(1) and 1302(3) to 1302(6) have beenrescaled to be smaller than the corresponding streams 1302(1) and1302(3) to 1302(6) shown in FIG. 13A, and have been repositioned in twocolumns along the left and right sides of the display surface 116.

In one embodiment, the movement and resealing operations shown in FIGS.13A and 13B are triggered by a user by selecting one of the projectedvideo streams 1302 (e.g., video stream 1302(2)) when the streams 1302are in their home positions (shown in FIG. 13A). In one embodiment, oneof the streams 1302 is selected by a user with user interface device1218, such as by pushing a key on a keyboard, or by selecting one of thestreams 1302 with a mouse device, and the streams 1302 are automaticallyrepositioned and rescaled by processing system 101. The location andsize of the projected video stream 1302(2) shown in FIG. 13B representsa “zoom” location and size according to one embodiment. The locationsand sizes of the projected video streams 1302(1) and 1302(3) to 1302(6)shown in FIG. 13B represents “hide” locations and sizes according to oneembodiment. The zoom location and size is used for a stream 1302 in oneembodiment when that stream 1302 is selected for emphasis by a user, andthe hide locations and sizes are used for streams 1302 in one embodimentwhen another stream 1302 has been selected for emphasis by a user.

FIG. 13C shows the transition of the six projected video streams 1302from the home locations and sizes shown in FIG. 13A to the zoom and hidelocations and sizes shown in FIG. 13B. In one form of the invention,when one of the streams 1302 shown in FIG. 13A is selected by a user(e.g., stream 1302(2)), the selected stream 1302(2) is gradually andcontinually scaled up in size to the zoom size as that selected stream1302(2) is also gradually and continually moved or slid across thedisplay surface 116 to the zoom location. At the same time the selectedstream 1302(2) is being moved and rescaled, the non-selected streams1302(1) and 1302(3) to 1302(6) are gradually and continually scaled downin size to the hide size as those non-selected streams 1302(1) and1302(3) to 1302(6) are also gradually and continually moved or slidacross the display surface 116 to their hide locations. During thetransition period between the stream positions and sizes shown in FIG.13A and the stream positions and sizes shown in FIG. 13C, one or more ofthe streams 1302 may cross over and at least partially overlap with oneor more of the other streams 1302 during the movement of these streams1302. In one embodiment, the streams 1302 appear semi-transparent sothat multiple overlapping streams 1302 can be viewed in the regions ofoverlap. In another embodiment, the streams 1302 appear opaque so thatonly one stream 1302 can be viewed in the regions of overlap.

In one embodiment, processing system 101 is configured to perform audiotransformations on one or more audio streams associated with one or moreof the projected streams 1302, such as fading audio in and out, andtransforming audio spatially over the speakers of display system 100. Inone embodiment, processing system 101 causes audio to be faded in for aselected stream 1302, and causes audio to be faded out for non-selectedstreams 1302.

In another embodiment of the present invention, processing system 101 isalso configured to allow a user to manually reposition and rescale oneor more of the projected streams 1302 using user interface 1218, andthereby allow a user to reposition the streams 1302 at any desiredlocations, and to rescale the streams 1302 to any desired size. Inaddition, in other embodiments of the invention, more or less than sixdifferent streams 1302 are simultaneously projected on surface 116 inany desired arrangement and size, and other emphasis options areavailable to a user (e.g., increasing the size of two streams 1302 whilemaking four other streams 1302 smaller). In another embodiment, ratherthan, or in addition to, relying on user input, processing system 101 isconfigured to automatically generate stream modification ortransformation parameters to modify the processed streams 1222 andcorrespondingly the projected streams 1302. For example, in one form ofthe invention, processing system 101 is configured to automaticallyposition and scale the streams 1302 based on the number of streams andwhere the streams 1302 are coming from (such as in a video conferencingapplication), or based on other factors.

Characteristics or properties of each stream 1302 may be transformedindependently by processing system 101. The properties that can betransformed according to one form of the invention include, but are notlimited to: (1) Two-dimensional (2D) screen space location and size; (2)three-dimensional (3D) location in the virtual screen space; (3)blending factors; (4) brightness and color properties; and (5) audioproperties. In one embodiment, properties of the streams 1302 aretransformed automatically by processing system 101 in response to anaction from a user, such as selecting one or more of the streams 1302with user interface device 1218. In another embodiment, a user interactswith processing system 101 via user interface device 1218 and display1220 to manually modify properties of one or more of the streams 1302.

In one embodiment, processing system 101 is configured to provideunconstrained transformations of the 2D and 3D properties of the streams1302. 2D transformations allow the streams 1302 to be slid around thedisplay surface 116, similar to how a window can be moved on a standardcomputer display, without any corresponding movement of the projectors112. The 3 D transformations include translations in depth, rotations,and scaling of the streams 1302.

Other types of image transformations are also implemented in otherembodiments. Streams 1302 that overlap on the surface 116 are blendedtogether by processing system 101 in one embodiment. Processing system101 is configured to allow a user to dynamically adjust blending factorsfor projected streams 1302. Processing system 101 is also configured toallow a user to dynamically adjust brightness and color characteristicsof projected streams 1302, allowing selected streams 1302 to behighlighted or deemphasized as desired. Processing system 101 is alsoconfigured to allow a user to perform cropping operations to selectedstreams 1302. In one embodiment, all transformations can be changeddynamically and independently for each stream 1302. The characteristicsof the streams 1302 can be changed in real time while still maintainingthe seamless nature of the display. In one form of the invention,processing system 101 is configured to combine one or more of thestreams 1302 with non-stream content, such as 3D geometry or models. Ina video conferencing application, for example, 2D video streams can beappropriately positioned by processing system 101 in a projected 3Dmodel of a conference room.

In one embodiment, the majority of the runtime computation of processingsystem 101 is performed by the GPUs 1214 and 1216, rather than by theCPUs 1210 and 1212. By performing most of the runtime computation on theGPUs 1214 and 1216, the CPUs 1210 and 1212 are left free to receive anddecompress multiple video and audio streams 1204. The GPUs 1214 and 1216perform color processing and conversion on the streams 1204, ifnecessary, such as converting from the YUV-4:2:0 format generated by anMpeg2 stream into RGB format for rendering. During geometric andphotometric calibration, geometric meshes 126 and photometric correctioninformation 128 are calculated as described above in Sections II andIII, and the geometric meshes 126 and photometric correction information128 are downloaded to the GPUs 1214 and 1216. At runtime, the geometricmeshes 126 and photometric correction information 128 do not need to berecalculated and can stay resident on the GPUs 1214 and 1216 for themultiple stream rendering.

Before the streams 1204 are geometrically mapped by GPUs 1214 and 1216,the geometric characteristics (including location) of the streams 1204can be transformed via a matrix multiply allowing any desiredtranslation, rotation, or scaling to be applied to the streams 1204. Thephotometric correction information 128 is then combined with the streams1204 by GPUs 1214 and 1216 to apply photometric correction and blendingin overlap regions. In one embodiment, photometric correction is appliedvia fragment shader programs running-on the GPUs 1214 and 1216. Forevery pixel that is to be displayed, the fragment program calculates thedesired RGB color. The GPUs 1214 and 1216 then use a gamma function tomap the pixel into the physical brightness space where the actualprojected values combine. Photometric correction is done in thisprojected light space before an inverse gamma function brings the colorvalues back to linear RGB.

The runtime processing performed by processing system 101 according toone form of the invention consists of acquiring streams 1204 from one ormore sources, preparing the streams 1204 for presentation, and applyingthe geometric meshes 126 and photometric correction information 128calculated during calibration. In one form of the invention, thereal-time processing and rendering is implemented using streamprocessing software modules 1208 in a multimedia framework 1206 (FIG.12). In one embodiment, multimedia framework 1206 is the “Nizza”framework developed by Hewlett-Packard Laboratories. The Nizza frameworkis described in Tanguay, Gelb, and Baker, “Nizza: A Framework forDeveloping Real-time Streaming Multimedia Applications”, HPL-2004-132,available at http://www.hpl.hp.com/techreports/2004/HPL-2004-132.html,which is hereby incorporated by reference herein. In another embodiment,a different multimedia framework 1206 may be used, such as DirectShow,the Java Media Framework, or Quicktime.

The Nizza framework is a software middleware architecture, designed forcreating real-time rich media applications. Nizza enables complexapplications containing multiple audio and video streams to run reliablyin real-time and with low latency. In order to simplify the developmentof applications that fully leverage the power of modem processors, Nizzaprovides a framework for decomposing an application's processing intotask dependencies, and automating the distribution and execution ofthose tasks on a symmetric multiprocessor (SMP) machine to obtainimproved performance. Nizza allows developers to create applications byconnecting media processing modules, such as stream processing modules1208, into a dataflow graph.

FIG. 14 is a diagram illustrating a dataflow graph showing theconnections of stream processing modules 1208 according to oneembodiment of the present invention. The stream processing modules 1208simultaneously receive six audio and video streams 1204, and process thestreams 1204 to generate processed streams 1222 (FIG. 12) to beprojected by projectors 112. Connections between the software modules1208 indicate where a stream leaves one module and enters a subsequentmodule for processing. Stream processing begins at the top of the graphshown in FIG. 14 and flows down through the modules 1208 at the bottomof the graph. As shown in FIG. 14, stream processing modules 1208include six network receiver software modules 1402(1)-1402(6), six audiodecompression software modules 1404(1)-1404(6), six video decompressionsoftware modules 1406(1)-1406(6), six gain control software modules1408(1)-1408(6), projectors software module 1410, and six speakersoftware modules 1412(1)-1412(6).

Network receiver software modules 1402(1)-1402(6) simultaneously receivesix audio and video streams 1204 (FIG. 12). In one embodiment, the audioand video streams 1204 received by network receiver software modules1402(1)-1402(6) are Mpeg2 transport streams. The network receivermodules 1402(1)-1402(6) each receive a different Mpeg2 transport stream,and reassemble the stream to generate a compressed audio stream and acompressed video stream. The compressed audio streams generated bynetwork receiver modules 1402(1)-1402(6) are provided to audiodecompression modules 1404(1)-1404(6), which decompress the receivedaudio streams, and provide the decompressed audio streams to gaincontrol modules 1408(1)-1408(6). Gain control modules 1408(1)-1408(6)perform a gain operation on the received audio streams so that audiofades in and out based on which stream is selected or emphasized asdescribed above with respect to FIGS. 13A-13C. The gain adjusted audiostreams generated by gain control modules 1408(1)-1408(6) are providedto speaker modules 1412(1)-1412(6), which control speakers of thedisplay system 100.

The compressed video streams generated by network receiver modules1402(1)-1402(6) are provided to video decompression modules1406(1)-1406(6), which decompress the streams into YUV-4:2:0 imagestreams. The YUV-4:2:0 image streams from the video decompressionmodules 1406(1)-1406(6) are provided to projectors software module 1410.Projectors software module 1410 performs geometric and photometricprocessing on the six received image streams as described above inSections II and III, and combines the streams into four processedstreams 1222 for projection by four projectors 112.

Software modules 1208 can process streams 1204 from many differentsources, including compressed Mpeg2 video streams from prerecordedsources such as DVDs and high-definition video, as well as live videosources compressed by remote Nizza modules or other video codecs. Othervideo or image sources can also be used to provide streams 1204 tosoftware modules 1208, including Firewire cameras, Jpeg image sequences,BMP image sequences, PPM sequences, as well as other camera interfaces.

FIG. 15 is a diagram illustrating a method 1500 of displaying multipleimage streams according to one embodiment of the present invention. Inone embodiment, image display system 100 is configured to perform method1500. At 1502, a plurality of image streams 1204 are provided toprocessing system 101. In one embodiment, each image stream 1204 in theplurality includes different image content than the other image streams1204 in the plurality. At 1504, the plurality of image streams 1204 areprocessed by processing system 101, thereby generating at least oneprocessed image stream 1222. At 1506, the at least one processed imagestream 1222 is projected onto a non-planar surface 116 with at least oneprojector 112, thereby generating a plurality of different projectedimage streams 1302 at a corresponding plurality of different positionson the non-planar surface 116, wherein each of the projected imagestreams 1302 corresponds to one of the image streams 1204 in theplurality of image streams 1204.

Although specific embodiments have been illustrated and describedherein, it will be appreciated by those of ordinary skill in the artthat a variety of alternate and/or equivalent implementations may besubstituted for the specific embodiments shown and described withoutdeparting from the scope of the present invention. This application isintended to cover any adaptations or variations of the specificembodiments discussed herein. Therefore, it is intended that thisinvention be limited only by the claims and the equivalents thereof.

1. A method of displaying multiple image streams, the method comprising:providing a first plurality of image streams; processing the pluralityof image streams, thereby generating at least one processed imagestream; and projecting the at least one processed image stream onto anon-planar surface with at least one projector, thereby generating aplurality of different projected image streams at a correspondingplurality of different positions on the non-planar surface.
 2. Themethod of claim 1, wherein the non-planar surface is a non-planardevelopable surface.
 3. The method of claim 1, wherein the projectedimage streams are positioned on the non-planar surface in anon-overlapping or substantially non-overlapping arrangement, whereineach image stream in the first plurality includes different imagecontent than the other image streams, and wherein each of the projectedimage streams corresponds to one of the image streams in the firstplurality of image streams.
 4. The method of claim 1, and furthercomprising: receiving selection information from a user identifying oneof the projected image streams.
 5. The method of claim 4, and furthercomprising: automatically moving the identified projected image streamto a different position on the non-planar surface in response toreceiving the selection information.
 6. The method of claim 4, andfurther comprising: automatically moving all of the projected imagestreams to different positions on the non-planar surface in response toreceiving the selection information.
 7. The method of claim 4, andfurther comprising: automatically changing a size of at least one of theprojected image streams in response to receiving the selectioninformation.
 8. The method of claim 1, and further comprising: receivingstream modification information; and modifying at least one property ofat least one of the projected image streams in response to receiving thestream modification information.
 9. The method of claim 8, wherein theat least one property is a display position or display size of the atleast one projected image stream.
 10. The method of claim 8, wherein themodification of at least one property comprises an audio transformationof at least one audio stream associated with the at least one projectedstream.
 11. The method of claim 8, wherein the modification of at leastone property comprises at least one of a 3D transformation, a blendingchange, a brightness change, and a color change.
 12. The method of claim1, wherein the processing of the plurality of image streams comprisesgeometric processing and photometric processing.
 13. The method of claim1, wherein the at least one projector includes a plurality of projectorspositioned in a tiled arrangement.
 14. The method of claim 13, wherein aquantity of projectors in the plurality of projectors is different thana quantity of projected image streams in the plurality of projectedimage streams.
 15. A system for displaying images, comprising: at leastone processing unit configured to process a received plurality of videostreams, thereby generating at least one processed video stream; atleast one projector for projecting the at least one processed videostream onto a display surface, thereby generating a plurality ofdifferent projected video streams at a corresponding plurality ofdifferent positions on the display surface; and wherein the at least oneprocessing unit is configured to cause movement of at least one of theprojected video streams on the display surface.
 16. The system of claim15, wherein the display surface is a non-planar developable surface. 17.The system of claim 15, wherein the processing unit is configured tocause repositioning and rescaling of at least one of the projected videostreams in response to selection information identifying one of theprojected video streams.
 18. The system of claim 15, wherein the atleast one projector includes a plurality of projectors positioned in atiled arrangement.
 19. The system of claim 18, wherein a quantity ofprojectors in the plurality of projectors is different than a quantityof projected video streams in the plurality of projected video streams.20. A method of generating a stream for projection onto a non-planarsurface, the method comprising: receiving a plurality of different videostreams; processing the plurality of video streams, thereby generatingat least one processed video stream, wherein the at least one processedvideo stream is configured to produce a plurality of different projectedvideo streams at a corresponding plurality of different positions on thenon-planar surface when projected; and modifying the processed videostream in response to stream modification information such thatprojection of the modified processed video stream causes at least one ofa 2D transformation, a 3D transformation, a brightness change, and acolor change of at least one of the projected video streams.